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1. Introduction 



The Study of Monitoring and Improvement Practices under the Individuals with Disabilities 
Education Act (IDEA) examined how states monitored the implementation of IDEA by local 
special education and early intervention services programs. State monitoring and improvement 
practices in 2004-05 and 2006-07 were the focus of the study. 

Prior to the mid-1970’s, Congress found that the special education needs of the 8 million 
children with disabilities were not being met (Education of All Handicapped Children Act of 
1975) (P.L. 94-142). To meet the educational needs of these children, Congress, in 1975, enacted 
the Education for All Handicapped Children Act (EHA). In EHA, Congress declared that the 
purpose of the act was to: 

“...assure that all handicapped children have available to them, ...a free appropriate 
public education which emphasizes special education and related services designed to 
meet their unique needs, to assure that the rights of handicapped children and their 
parents or guardians are protected, to assist States and localities to provide for the 
education of all handicapped children, and to assess and assure the effectiveness of 
efforts to educate handicapped children.” (P.L. 94-142, §3, (c)). 

A key amendment to the act was passed in 1986 when Congress provided funds for states to 
serve infants and toddlers with disabilities. In 1990, EHA was renamed the Individuals with 
Disabilities Education Act (IDEA), and the term “handicapped child” was replaced with the term 
“child with a disability.” Part B of the statute covers children and youth ages 3 through 2 1 with 
disabilities, while Part C covers infants and toddlers ages birth through 2 with disabilities and 
their families. State Part B programs are administered by state education agencies (SEAs). States 
may choose the SEA or another agency to administer Part C. 

Under P.L. 94-142, Congress gave states the responsibility for ensuring that the provisions of the 
law were being carried out at the local level. Part C agencies were given similar responsibilities 
when Congress authorized the program in 1986. At the federal level, the U.S. Department of 
Education’s (ED) Office of Special Education Programs (OSEP) within the Office of Special 
Education and Rehabilitative Services (OSERS) was the agency designated in IDEA as 
responsible for ensuring that states carry out the provisions of the law. Specific responsibility lies 
with OSEP’s Monitoring and State Improvement Planning Division (MSIP). 

As the law changed over the years, so did the way that OSEP approached its monitoring 
responsibilities. The greatest change was the shift that began in the late 1980s from a focus on 
compliance with the law’s rules and regulations to an added focus on improving educational 
results and functional outcomes for children with disabilities (Ahearn 1995). This change 
became a formal element in the 1997 and 2004 amendments to IDEA for both OSEP and the 
states. The 2004 amendments also provided specific directions to OSEP and states on focus 
areas, measurable indicators, and enforcement actions to be used in monitoring compliance with 
IDEA. 
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Study of States’ Monitoring Under IDEA 



In 2004, OSEP contracted with Westat to conduct a 5-year study of states’ monitoring and 
improvement practices under IDEA. Until 2004, there had been no independent and systematic 
examination of monitoring systems across the states. In OSEP’s view, such an examination of 
monitoring systems could infonn its efforts to provide monitoring guidance to states and, thus, 
enable it to better carry out its responsibilities under the act. 

The study had the following three objectives: 

1. To provide a description of the nature and scope of states’ monitoring systems. 

2. To describe states’ monitoring systems at two points in time. 

3. To create a framework to describe state monitoring systems. 

In 2006, the study was transferred from OSEP to the National Center for Special Education 
Research (NCSER) within the Institute of Education Sciences (IES). However, the primary 
purpose of this report remains to provide OSEP with information on states’ Part B and Part C 
monitoring systems. 



Report Overview 

This report presents the findings from the study. Chapter 2 provides a brief history of the IDEA 
monitoring requirement as it relates to this study; chapter 3 provides a discussion of how a 
framework for state monitoring was developed; chapter 4 describes the methods for obtaining 
information about states’ Part B and Part C monitoring systems; chapter 5 presents the findings 
from the site visit data collection efforts; 1 and chapter 6 presents a summary of the key findings 
from the study. 

A list of the frequently used acronyms that appear in this report can be found in appendix A. 



1 The first data collection effort for this study was a mail survey of all Part B and Part C state agencies in the 50 
states and the District of Columbia. The findings from the survey were reported in Garrison-Mogren et al. 2008. 
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2. History of Monitoring and Improvement Practices 



IDEA and its accompanying regulations shape state monitoring and improvement systems by 
providing an agenda for their monitoring efforts. The mandate has changed over the years and 
has necessitated that states change their monitoring systems to meet new oversight 
responsibilities under the act. OSEP creates monitoring priorities and procedures based on the 
requirements of IDEA to ensure that states are carrying out their responsibilities under the law. 
OSEP’s implementation of each new congressional mandate, as well as changes it makes in its 
procedures to oversee states’ monitoring efforts, influence what states actually monitor and how 
they monitor (Ahearn 2007; Lee 2004; Tschantz 2002a, 2000b). To meet the objectives of this 
study, it was important for us to examine the historical context for the development of state 
monitoring systems. This chapter provides an overview of this historical context. 

For this study, the project team categorized development and progression of legislative 
influences on state monitoring and OSEP’s related oversight activities into four phases: the early 
years of monitoring, OSEP’s monitoring refocus, the influence of the 1997 amendments to 
IDEA, and the influence of the 2004 amendments to IDEA. Figure 1 presents a timeline of major 
federal legislation and OSEP’s oversight procedures and requirements for state monitoring from 
1975 to 2007. These phases delimit approaches to monitoring at both the federal and state levels. 
During each phase, monitoring foci, practices, and procedures changed based on changes in 
IDEA, or changes in OSEP’s monitoring procedures. The influence of the 1997 and the 2004 
amendments is described more comprehensively as they provided the context during which data 
were collected for this study, that is, for 2004-05 and 2006-07. 



The Early Years of Monitoring 

EHA required that, before funding is committed for a new year, SEAs must (1) ensure 
compliance with the procedural requirements of the law and (2) provide oversight of programs 
providing special education services under Part B in the state (EHA 1975). 

Under P.L. 94-142, SEAs and the U.S. Office of Education, Bureau of Education for the 
Handicapped (BEH, the predecessor to OSEP) were given interrelated and parallel 
responsibilities to ensure that the purposes of the act were implemented. With its passage, SEAs 
were required to establish procedures to ensure that local education agencies (LEAs) and other 
entities (e.g., correctional facilities, schools for students with hearing impairments, and others) 
provided a free appropriate public education (FAPE) to children with disabilities. P.L. 94-142 
referred to this responsibility as “general supervision.” The general supervision provision also 
required SEAs to perform various other administrative functions, such as distribution of EHA 
monies to LEAs. To ensure that SEAs were carrying out these responsibilities, BEH 
implemented oversight procedures that included reviewing state plans, conducting program 
reviews in states, processing complaints, and developing waiver procedures for funding when 
various aspects of the law could not be met, such as when the state had a shortage of certified 
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teachers (U.S. Department of Health, Education, and Welfare 1979, p. 72). The law also 
specified that BEH provide technical assistance to help SEAs carry out their responsibilities. The 
law did not, however, fully define or specify the procedures states should use for monitoring. 
Thus, early state Part B monitoring efforts were inconsistent and fragmented (Rostetter 1988). In 
addition, the law provided little guidance for BEH enforcement of state compliance with these 
requirements. 

In the late 1970s, BEH initiated a program review process to oversee SEAs’ efforts. Key aspects 
of the program review process involved review and approval of state plans, verification of the 
implementation of corrective action plans (CAPs), and periodic on-site visits to assess states’ 
policies and procedures to identify areas of noncompliance with the law. By 1980, all SEAs had 
monitoring and improvement systems in place. However, BEH found that only 5 of the 21 SEAs 
evaluated were in full compliance with the law’s monitoring provisions (U.S. Department of 
Education 1980). In the 1980s and early 1990s, OSEP often changed its monitoring procedures, 
and SEAs frequently changed the content or processes of their monitoring systems to respond to 
changes in judicial interpretations of the act, other federal legislation, amendments to the act, 
congressional inquiries, policy initiatives, and OSEP’s monitoring findings. State monitoring 
systems differed from one another as a result; some states followed OSEP’s lead in making 
changes, while others did not (Ahearn 1997). The focus, however, remained on ensuring 
compliance with explicit requirements noted in Part B of IDEA. 

In 1986, during the reauthorization of IDEA, a new program (initially under Part H, now under 
Part C) was enacted to assist states in developing and implementing statewide systems of early 
intervention services for eligible infants and toddlers and their families. One of the required 
aspects of this statewide system was “a single line of responsibility in a lead agency designated 
or established by the Governor for carrying out — (A) the general administration, supervision, and 
monitoring of programs and activities receiving assistance under section 673 to ensure 
compliance with this part,...” (P.L. 99-457, §1435(a)(10)). IDEA gave states up to 7 years to 
fully implement Part C. 



OSEP’s Monitoring Refocus 

In the early 1990s, with the help of multiple stakeholders, OSEP redesigned its approach to 
overseeing and guiding states’ monitoring for Part B implementation (U.S. Department of 
Education 1993). The need for a strong accountability system and those requirements that most 
directly related to improving outcomes for students with disabilities were emphasized. The 
redesign included (1) extensive and diverse stakeholder input into the monitoring process; 
(2) effective methods for ensuring compliance with Part B, with an emphasis on those 
requirements closely linked to outcomes for students with disabilities; (3) prompt identification 
and correction of noncompliance; and (4) corrective action requirements and strategies that lead 
to improved access and outcomes for students with disabilities (U.S. Department of Education 
1995, p. 131). OSEP used this modified program review process to oversee SEAs from the 
1994-95 through 1997-98 school years. In school year 1997-98, Part C lead agencies became a 
full participant in OSEP’s monitoring cycle, with OSEP using the same basic process to oversee 
states’ compliance monitoring that was used for Part B. 
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The 1997 Amendments to IDEA 

The 1997 amendments to IDEA required that states develop a set of performance goals and 
indicators on which to judge their progress in meeting their goals for children with disabilities. 
States were also required to report progress on their performance goals and indicators to OSEP 
and the public every 2 years in a Biennial Progress Report. The 1997 amendments also included 
a provision for State Improvement Grants; the purpose of these competitive grants is to help 
states meet their performance goals. 

In 1998, OSEP convened stakeholder workgroups to inform revisions in its approach to 
overseeing and assisting with states’ monitoring, based on the new requirements in the 1997 
amendments. This approach, entitled Continuous Monitoring Improvement Process (CIMP), had 
several themes, including (1) continuity, (2) partnerships with stakeholders, (3) state 
accountability, (4) self-assessment, (5) data-driven decisionmaking, (6) public process, and (7) 
technical assistance (U.S. Department of Education 1999, p. IV-46). OSEP used CIMP to 
monitor states from the 1998-99 through 2001-02 school years. There was no requirement, 
however, that states adopt this approach in monitoring local agencies. 

OSEP conceptualized the CIMP model as a continuous cycle composed of the following 
activities: (1) self-assessment, (2) validation planning, (3) validation data collection, (4) 
improvement planning, (5) implementation of improvement strategies, (6) verification and 
consequences, and (7) review and revision of self-assessment. Under CIMP, OSEP required that 
states conduct state-level self-assessments for Part B and Part C, guided by stakeholder steering 
committees. Each state’s self-assessment used data on its perfonnance goals and indicators to 
evaluate strengths and weaknesses of the state’s special education and early intervention 
services. The states’ steering committees then developed State Improvement Plans based on self- 
assessment findings and areas in need of improvement. OSEP provided technical assistance to 
help states implement improvement strategies, regardless of whether the states used CIMP or 
another approach to monitor their local programs, to achieve compliance. If a state did not make 
the necessary improvements, OSEP could impose enforcement actions (U.S. Department of 
Education 2000, p. IV-25). 

In 2000, OSEP modified certain aspects of CIMP as a result of its implementation experience 
(U.S. Department of Education 2002a, p. IV-39) and a report from the National Council on 
Disability (NCD 2000) that concluded that every state in the country was out of compliance with 
IDEA. The modifications by OSEP reiterated several themes previously identified. These themes 
were state accountability, enhanced involvement by public stakeholders, the use of infonnation 
obtained from state self-assessments, the use of state data provided to ED under IDEA, 
improvement, technical assistance and support, and consequences (U.S. Department of 
Education 2002a, pp. IV-39-IV-41). OSEP designated a small number of IDEA requirements for 
state monitoring judged to have the strongest relationship with positive outcomes for infants, 
toddlers, and children with disabilities (U.S. Department of Education 2001, p. IV-4). For Part C, 
these were (1) general supervision, (2) child find and public awareness, (3) early intervention 
services in natural environments, (4) family-centered systems of services, and (5) early 
childhood transition. For Part B, these priority areas were (1) parent involvement, (2) FAPE, (3) 
secondary transition, and (4) general supervision. 
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After meeting with a diverse group of stakeholders and using findings from the President’s 
Commission on Excellence in Special Education (U.S. Department of Education 2002b), OSEP 
began developing a process to target resources on high-priority areas that were judged to have 
the highest likelihood of improving outcomes for children with disabilities. This new approach, 
influenced by the passage of No Child Left Behind, combined CIMP with the targeting of 
resources on specific areas of IDEA implementation (Lee 2005). This approach, Continuous 
Improvement Focused Monitoring System (CIFMS), was implemented in 2003 and sought 
evidence of improved outcomes through the collection of data on a set of specified perfonnance 
measures. The process had a four-part accountability strategy that included ensuring that states’ 
monitoring systems were in compliance with IDEA, attending to high-risk states (those at high 
risk for compliance or financial and/or management failure), supporting states in implementing 
continuous improvement strategies, and focusing assistance on states with low performance on 
the established high-priority areas (Lee 2003). In 2004, OSEP funded the National Center on 
Special Education Accountability Monitoring (NCSEAM) to provide technical assistance to 
states in developing focused monitoring systems. 



The 2004 Amendments to IDEA 

In December 2004, the Individuals with Disabilities Education Improvement Act of 2004 was 
signed into law, once again making significant changes to federal and state monitoring 
responsibilities. Up to this point, CIMP and CIFMS were not requirements of the law and, 
although some states incorporated the principles of OSEP’s two monitoring approaches into their 
systems, others did not (Tschantz 2002a). The 2004 amendments codified CIFMS to be used by 
states to monitor local programs either as a part of or in addition to their monitoring systems. The 
amendments strengthened the 1997 emphasis on child outcomes while also requiring that 
resources be targeted to the collection of data associated with specific areas of IDEA 
implementation (Lee 2004). Section 616 of the reauthorized IDEA established the primary 
purpose of federal oversight and state monitoring as follows: 

• improving educational outcomes and functional outcomes for all children with 
disabilities and 

• ensuring that states meet the program compliance requirements with a particular 
emphasis on those requirements that are most closely related to improving 
educational outcomes for children with disabilities. 

The amendments required the Secretary of Education to oversee states’ monitoring efforts and 
required each state to monitor local programs using quantifiable and qualitative indicators of key 
performance measures designated by the new amendments and/or OSEP. The amendments 
required states to submit a 6-year State Performance Plan (SPP) within a year after the statute 
was signed into law (December 2004). The framework of the SPP developed by OSEP required 
that states provide baseline data on each specified indicator, as well as 6 years of projected 
targets for each indicator. The Part B SPP includes 20 indicators, and the Part C SPP includes 14 
indicators. 



7 




Each state is required to collect data on all SPP indicators and to submit an Annual Performance 
Report (APR) to OSEP. The first APRs for Part B and Part C were due February 1, 2006. APRs 
are to be posted on each state’s website, reported to the media, and distributed through public 
agencies. 

In addition to the SPPs and APRs, IDEA 2004 also required that the Secretary of Education 
develop and implement a determinations process. Each year, beginning in 2007, OSEP reviews 
each state’s APR, information from oversight visits, audit results, and other public information to 
determine if the state (1) meets the requirements and purposes of IDEA, (2) needs assistance in 
implementing the requirements of IDEA, (3) needs intervention in implementing the 
requirements of IDEA, or (4) needs substantial intervention in implementing the requirements of 
IDEA. The statute also included a graduated set of required enforcement activities (e.g., 
requiring compliance agreements, withholding funds) that relate to specific detenninations made 
by OSEP for states. Similarly, states were required to make one of the same four detenninations 
annually about the performance of each local school district or local early intervention program. 
In making detenninations, states must use performance indicator compliance; an assessment of 
the validity, reliability, and timeliness of the data submitted by the LEAs/early intervention 
services (EIS) programs; unconected noncompliance data from other sources; and any audit 
findings to make these determinations. States must implement the same enforcement actions as 
OSEP. Some states use additional enforcement actions based on their own laws and regulations 
(NECTAC 2007). 

Summary for History of Monitoring and Improvement Practices 

State monitoring of local special education and early intervention programs is a critical element 
in ensuring that the purposes of IDEA are being met. In 1975, P.L. 94-142 required that states 
assume this responsibility, and OSEP was given the responsibility to monitor states’ compliance 
with the law, including state monitoring systems. As such, state and OSEP monitoring and 
accountability efforts are necessarily intertwined. Moreover, as IDEA has been amended since 
1975, states have been given new monitoring responsibilities. In turn, OSEP created new 
monitoring procedures over the same period to ensure accountability in states’ monitoring of 
IDEA. For the first time, the 2004 amendments provided specific and parallel directions to OSEP 
and states on focus areas, measurable indicators, and enforcement actions to be used in 
monitoring compliance with IDEA. 

The history of state and federal monitoring provides a key to understanding the nature and scope 
of state monitoring systems examined for this study. It also provided a set of activities that 
assisted in the development of a framework for monitoring for this study that is described in 
chapter 3. 
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3. Development of a Framework for State Monitoring 
Systems 



One objective of the study was to develop a framework to describe state monitoring systems. The 
purpose of this framework was to provide OSEP and other stakeholders a greater understanding 
of the variation in states’ Part B and Part C monitoring systems. 

This chapter discusses the rationale and expectations for this framework, describes the 
development process, and presents the resulting framework. 



Rationale and Expectations 

As reviewed in the prior chapter, changes in the statute and regulations and variations in the way 
that states have responded to guidance from OSEP over the years have produced a diverse set of 
monitoring systems implemented by states. To date, there is no generally accepted standard for 
what a state monitoring system should look like. IDEA did not impose any particular monitoring 
approach on the states, nor did it empower OSEP to do so. It has also been noted in the literature 
that there is high variability in monitoring system designs across states (e.g., Comstock-Galagan 
and O’Connell 2002; Rostetter 1988). Given these circumstances, the project team decided to 
develop an overarching framework that would incorporate important features of state monitoring 
systems, as determined through analysis of varied documentary sources and consultation with 
experts on IDEA and monitoring practices. This framework then guided the development of data 
collection methods and tools, which are described in chapter 4. 

The framework serves as a means for organizing the description of state monitoring systems and 
ensuring that important details about the systems are systematically documented. Readers should 
be aware that this framework has not been validated or evaluated regarding whether the adoption 
of the framework results in an improved monitoring system and, therefore, conclusions drawn 
from the findings of the study should be limited accordingly. In particular, the framework should 
not be viewed as a standard that states should meet, nor should the reader infer that states that 
more closely emulate this framework have better monitoring systems or better outcomes. 



Process for Developing the Framework 

The process for developing the framework generally followed three steps. Each of these steps is 
described in more detail below. 

Literature Review 

As the first step in the development of the framework, project staff collected relevant literature 
on provisions of IDEA and state monitoring activities. In addition to published articles, the 
project staff also compiled OSEP policy memoranda; assembled various sections from the IDEA 
annual reports to Congress on OSEP monitoring procedures, for example, CIMP and CIFMS; 
reviewed materials from NCSEAM, for example, the NCSEAM Focused Monitoring 
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Implementation Checklist (NCSEAM 2004); and examined a large number of unpublished 
documents that were made available by OSEP, for example, materials used by various OSEP 
presenters at meetings. 

The project team reviewed all of these materials to identify key activities that needed to be 
considered in the development of the framework. For example, IDEA requires that states identify 
issues related to noncompliance with the law and have mechanisms in place for addressing these 
issues. Furthermore, IDEA 2004 places additional emphasis on improving educational and 
functional outcomes for all children with disabilities. As another example, the NCSEAM 
Focused Monitoring Implementation Checklist includes items related to stakeholder involvement 
in the monitoring process; data collection, analysis, and reporting; and examination of local 
performance on selected indicators. As a result of this literature review, the project team was 
able to identify a number of activities that seemed to be relevant to states’ monitoring efforts. 

Development of Framework Components 

The second step in the development process was taking the key activities that were identified 
during the literature review and organizing them into distinct sets of tasks. These distinct sets of 
tasks became the components of the framework. To do this, the project team reviewed all of the 
various activities and grouped interrelated activities together. For instance, as noted above, IDEA 
requires states to have procedures in place to identify issues related to noncompliance. Therefore, 
the project staff came to the conclusion that one component of the framework needed to be 
devoted to the identification of such issues. The project team then discussed what kinds of 
activities would be needed by the state in order to identify issues. Activities identified from the 
literature review, such as collecting and analyzing data and assessing local performance, seemed 
to be related to this task, and, therefore, they were incorporated into this component. As another 
example, once a state identified an issue, it needed to have mechanisms in place to address the 
issue. Therefore, the project team concluded that one or more components of the framework 
needed to be devoted to addressing identified issues. Again, the project team reviewed the 
findings from the literature review and pulled out activities that seemed to be related to 
addressing identified issues, such as correcting noncompliance issues and improving outcomes 
for children with disabilities. Using this process, the project team came up with a number of 
preliminary framework components. 

The project team presented the preliminary framework components to the study’s Advisory 
Panel, which consisted of a broad representation of stakeholders with expertise in program 
evaluation; state monitoring practices; parent advocacy; state Part C and Part B administration; 
technical assistance; special education law, regulations, and policy; and early intervention and 
preschool special education. The project team discussed with the Advisory Panel the preliminary 
framework components and the potential relationships between them. One of the key pieces of 
feedback from the Advisory Panel was in regard to the relationships between the components. 
The Advisory Panel felt it was critical to present the relationships between the components as 
cyclical rather than sequential. That is, the components that came later in the sequence provided 
information that affected the implementation of components that came earlier in the sequence. 
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Therefore, the framework and its components were further revised to reflect the cyclical nature 
of monitoring, as well as other comments from the Advisory Panel. Additional feedback on the 
revised framework components was obtained from the Advisory Panel and from OSEP on 
several occasions. The project team further refined the framework components based on this 
feedback before finalizing them. 

Defining Key Characteristics and Elements 

The third step involved identifying key characteristics thought to be important to states’ 
monitoring efforts for each component of the framework. The project team initially drew upon 
the Joint Committee on Standards for Educational Evaluations (JCSEE) standards because they 
provide universal ideals for evaluating the adequacy and soundness of evaluations that could be 
applied to evaluation efforts, such as state monitoring efforts. Although the JCSEE refers to 
“standards,” this study uses the tenn “key characteristic” because, as noted above, the framework 
has not been tested or validated. 

There are four categories of JCSEE standards: utility, feasibility, propriety, accuracy. Each 
category is described briefly below, focusing on themes that particularly informed the key 
characteristics of each component of the framework. 

1 . The utility standards address the extent to which an evaluation meets the needs of its 
intended audience. The utility standards “guide evaluations so that they are 
informative, timely, and influential” (JCSEE 1994, p. 5). They address the extent to 
which evaluators are knowledgeable of their audience and respond to the audience’s 
needs and concerns. The standards also address such issues as the credibility of 
evaluators, the scope and clarity of reports, and how reports are disseminated. 

2. The feasibility standards address issues related to whether an evaluation is realistic 
and sensible (i.e., practical). The feasibility standards cover such issues as whether 
the procedures are practical and cost effective. 

3. The propriety standards focus on ethical and legal issues. They address issues such as 
the rights of human subjects, complete and fair assessment, and disclosure of 
findings. 

4. The accuracy standards address the technical adequacy of an evaluation. While the 
previous standards offer important context for conducting evaluations, the accuracy 
standards assess the soundness of the methodology. The standards cover such issues 
as program documentation, description of purposes and procedures, the need for 
defensible information, reliability and validity of information, methods of analysis, 
and justified conclusions. 

Using the JCSEE standards and the other sources described earlier as starting points, the project 
team defined key characteristics for each component of the framework. For example, the JCSEE 
utility standards, the principles of CIFMS, and the NCSEAM Focused Monitoring 
Implementation Checklist all emphasize the importance of including and obtaining input from 
stakeholders. Therefore, the project team developed key characteristics related to the inclusion of 
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stakeholders in various monitoring activities. As another example, the JCSEE accuracy standards 
include items related to collecting valid and reliable data and reporting on findings. The 
NCSEAM Focused Monitoring Implementation Checklist also has items related to data 
collection and drafting reports. Therefore, the project staff used these items to develop key 
characteristics related to data collection, analysis, and reporting. 

The key characteristics were reviewed by the project team and Advisory Panel multiple times 
and revised as needed to ensure that they were capturing important state monitoring activities 
associated with each component of the framework. 

As a final step, the project team developed measurable elements that could be used during the 
site visits to determine whether each key characteristic was present in states’ Part B and Part C 
monitoring systems. Again, as part of the process, the Advisory Panel reviewed the elements 
until there was consensus on the elements for each key characteristic. The project team then used 
this input to finalize the elements. 



Framework Components, Key Characteristics, and Elements 

As discussed above, the development process led to a framework that includes (1) components, 
(2) key characteristics, and (3) elements. Figure 2 presents an overview of the framework, 
including its components and the interactions among them. 

A component represents a set of distinct and coherent tasks, each with its own unique purpose 
and focus related to monitoring IDEA implementation. As figure 2 shows, there are five distinct 
but interconnected components, including (1) Identification of Problems, (2) Problem 
Investigation, (3) Corrective Action and Enforcement, (4) Improvement Planning and 
Implementation, and (5) Reassessment. Together, the components and their interrelationships 
form the structure of the framework. 

The key characteristics define the components and identify aspects that are thought to be 
important to the component being carried out. For example, under Problem Identification, one 
key characteristic is that a formal process exists for identifying indicators and setting targets. 

The elements spell out what signifies the presence of each key characteristic. For example, when 
identifying indicators under Problem Identification, one element specifies that indicators are 
based on stakeholder input. 

Below, each component of the framework is described, and the list of the key characteristics 
associated with that component is presented. All the elements are not presented in this chapter 
due to space constraints, but the complete list of key characteristics and elements can be found in 
appendix B. 
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Figure 2. 



Framework for state monitoring 




Component: Problem Identification 

Problem Identification compares perfonnance (e.g., on a specific indicator) to an expectation of 
perfonnance (e.g., a target established for that indicator). In the context of state monitoring, it refers 
to assessing, documenting, and verifying that IDEA is being implemented as intended. Problem 
Identification can potentially alert states to problems that exist at the state or local levels. Problem 
Identification serves to infonn the state about perfonnance on targets that may indicate a failure to 
implement the law as intended or achieve satisfactory outcomes, with the assumption being that 
falling below a target requires action at the state or local level. 
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Problem Identification has three subcomponents, including: 1) Indicator and Target Setting, 2) 
Indicator Data Collection and Analysis, and 3) Problem Detection. Each is described below. 

1) Indicator and Target Setting 

Indicators and targets are a primary focus of Problem Identification and set expectations for the 
monitoring system performance. They represent key areas that the state, including stakeholders, 
believes are most important to monitor in order to evaluate the performance of the early 
intervention or special education programs. Since IDEA specifies procedures and regulations that 
states must follow when providing early intervention or special education services, a state 
monitoring system must include indicators that measure compliance with the law. Other 
indicators should address outcomes (e.g., graduation rates, suspension/expulsion rates, the 
proportion of children served under Part C) that can also provide information about whether 
services are being provided as intended. States must also set targets or acceptable levels of 
performance for their indicators. 

Four key characteristics pertain to Indicator and Target Setting: 

• Al. Stakeholder Participation : Stakeholders are informed and contribute throughout 
the indicator and target setting process. 

• A2. Indicator Identification : There is a process to identify indicators. 

• A3. Measurable Indicators : Indicators are clearly written, logical, quantifiable, and 
represent valid measures of IDEA implementation. 

• A4. Rigorous and Realistic Targets : Selected targets are documented and reflect the 
state’s indicators. 

2) Indicator Data Collection and Analysis 

Indicator Data Collection and Analysis involves the collection, preparation, and analysis of data 
needed to assess perfonnance on the identified indicators. This process may include ongoing data 
collection efforts mandated by IDEA (e.g., the Section 618 data requirements) or new data collection 
efforts in response to stakeholder-identified indicators. Indicator Data Collection and Analysis 
involves collecting reliable and valid data in the most practical, efficient, and ethical ways. 

Six key characteristics pertain to Indicator Data Collection and Analysis: 

• Bl. Credibility of Data Collection Team(s) : The individuals responsible for collecting the 
data used to measure LEA/EIS program perfonnance on indicators possess the requisite 
technical competence, substantive knowledge, and experience. 

• B2. Protection of Stakeholders : The rights and welfare of those affected by the data 
collected to measure LEA/EIS program perfonnance on indicators (i.e., infants, children 
and youth with disabilities and their families, teachers, and other services providers) are 
protected. 
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• B3. Thoroughly Documented Data Collection Process : The process for indicator data 
collection is described in sufficient detail so that the utility, reliability, and validity of 
the information collected can be assessed. 

• B4. Accurate Data : Data collection instruments and procedures are reliable and valid 
for the purposes intended. The data collected are examined to ensure that errors are 
corrected or nonexistent. 

• B5. Analysis of Indicator Data : Indicator data are analyzed to detennine LEA/EIS 
program performance. 

• B6. Practical and Efficient Procedures : Indicator data collection procedures are 
practical and minimize burden on the LEA/EIS program, including limiting 
disruption of daily routines of schools or programs. 

3) Problem Detection 

Problem Detection involves comparing perfonnance on a specified indicator to the target established 
for that indicator. Without Problem Detection, states may not know where problems exist. When 
perfonnance fails to meet the target, a potential problem is identified. There may be valid reasons for 
suboptimal perfonnance, such as a target that was set too high, perfonnance that was not measured 
conectly, or data that were not collected properly. If an objective review by relevant stakeholders 
detennines that the entire process is valid, and perfonnance is deficient, then Problem Investigation is 
used to detennine why the problem exists. It should be noted that when perfonnance on a specific 
indicator meets or exceeds the target, then it is logical to review the indicators, targets, and data 
collection procedures to decide whether to continue as is, increase the target, or change the focus 
altogether. 

Four key characteristics pertain to Problem Detection: 

• Cl. Defensible Findings : The findings drawn from the problem identification process are 
adequately defended and supported by comparing LEA/EIS program perfonnance to 
specified targets. 

• C2. Complete Reporting on the State : A report completely describes the problems that 
are identified throughout the state regarding perfonnance on specified indicators and 
targets, so that essential infonnation can be provided and is easily accessible to 
stakeholders. 

• C3. Complete Reporting on LEA/EIS Programs : Reports prepared by the state 
completely describe the problems that are identified regarding LEA/EIS program 
perfonnance on specified indicators and targets, so that essential infonnation can be 
provided and is easily accessible to local stakeholders. 

• C4. Dissemination of Problem Identification Reports to Stakeholders : Problem 
identification reports are accessible and available to stakeholders. 
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Component: Problem Investigation 

Problem Investigation involves exploring why an identified problem exists and whether it is systemic 
throughout the state or isolated in specific LEAs/EIS programs. Problem Investigation may include 
collection of new data through focus groups, surveys, or interviews; analyses of existing data; or 
other techniques. According to the framework (see figure 2), Problem Investigations can lead the 
state to one of three actions depending on the outcome of the investigation: (1) a return to Problem 
Identification, (2) a move to Corrective Action and Enforcement, or (3) initiation of Improvement 
Planning and Implementation. Returning to Problem Identification may be required when the state 
explores the data more extensively and finds that the indicator(s) or target needs revisiting. 
Corrective Action results from Problem Investigations when individual records at the local level are 
found to be deficient. By law, these must be corrected. If a Problem Investigation reveals that failure 
to meet a target(s) is symptomatic of a larger concern (either in an LEA/EIS program or statewide), 
then Improvement Planning and Implementation is the next step. 

Eight key characteristics pertain to Problem Investigation: 

• Dl. Stakeholder Participation : Stakeholders are infonned and contribute throughout the 
problem investigation process. 

• D2. Credibility of Problem Investigation Team : The individuals responsible for 
investigating identified problems possess the requisite technical competence, substantive 
knowledge, and experience. 

• D3. Practical and Efficient Procedures : Data collection procedures for problem 
investigations are practical, internally consistent, and efficient while minimizing burden 
on the LEA/EIS program, including the disruption of daily routines of schools or 
programs. 

• D4. Thoroughly Documented Problem Investigations : The process used to investigate 
identified problems is described in sufficient detail so that the utility, reliability, and 
validity of the infonnation can be assessed. 

• D5. Accurate Data : Data collection instruments and procedures are reliable and valid for 
the purposes intended. The data collected are examined to ensure that errors are corrected 
or nonexistent. 

• D6. Analysis of Quantitative Data : The analytic techniques fit the quantitative data and 
the identified problems under investigation. 

• D7. Analysis of Qualitative Data: The analytic techniques fit the qualitative data and the 
identified problems under investigation. 

• D8. Defensible Findings: The findings from the problem investigation process are 
adequately documented, defended, and supported by data, so that the state can proceed 
with improvement planning or corrective actions. 
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Component: Corrective Action and Enforcement 

If Problem Investigation reveals a need for corrective action, the LEA/EIS program must make 
immediate changes to its documentation, procedures, or practices. The state agency must take 
corrective action when a procedural requirement in the law or a requirement to provide appropriate 
services has not been met in any individual (child) case. Corrective actions are defined in IDEA. For 
example, a state may require an LEA/EIS program to change its procedures for obtaining parent 
signatures on individualized education program/individualized family service plan (IEP/IFSP) if the 
documents are often found without parent signatures. Findings of noncompliance must be corrected 
as soon as possible and in no case later than 1 year after identification. CAPs are written to ensure 
that such corrections are made. If multiple findings of noncompliance occur within an LEA/EIS 
program, the state agency may consider the problem to be “systemic” and require Improvement 
Planning in addition to a CAP. The state lead agency may apply penalties when a local program does 
not make satisfactory progress on corrective actions within the specified period of time. These 
enforcement procedures may include withholding funds. Corrective Action and Enforcement 
includes only short-term efforts to alleviate a particular (past) problem identified by the state in a 
particular LEA/EIS program. According to the framework, Corrective Action and Enforcement leads 
to Reassessment to detennine the impact of these activities on the identified problems. The 
Corrective Action and Enforcement component of the monitoring model does not include efforts to 
prevent similar problems in the future. Such efforts are included instead in Improvement Planning. 

Three key characteristics pertain to Corrective Action and Enforcement: 

• El. Detailed Corrective Action Plans : CAPs address the identified problem(s) related to 
procedural noncompliance and are sufficiently detailed. 

• E2. Dissemination of Corrective Action Plans to Stakeholders : CAPs are accessible and 
available to local stakeholders. 

• E3. Enforcement of Corrective Action Plans : Enforcement procedures are in place to 
ensure compliance with needed corrective action. 

Component: Improvement Planning and Implementation 

Improvement Planning and Implementation includes two subcomponents: (1) Improvement Planning 
and (2) Improvement Plan Implementation. Each is described below. 

1) Improvement Planning 

Improvement Planning involves developing strategies and identifying resources for bringing about 
change to LEA/EIS programs and services; for example, reviewing past successes, unsuccessful 
efforts/activities, and evidence-based strategies found to be effective elsewhere and developing a 
strategic plan that includes specific goals and activities designed to correct systemic noncompliance 
and/or improve program outcomes. 
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Four key characteristics pertain to Improvement Planning: 



• FI. Stakeholder Participation : Stakeholders are infonned and contribute throughout the 
improvement planning process. 

• F2. Detailed Improvement Plans : Improvement plans address the identified problems and 
are sufficiently detailed. 

• F3. Defensible Improvement Plans : Improvement plans reflect research and evidence- 
based practices, identify support for implementation, and include oversight. 

• F4. Dissemination of Improvement Plans to Stakeholders : Improvement plans are 
accessible and available to stakeholders. 

2) Improvement Plan Implementation 

Improvement Plan Implementation refers to the complex interactions required to bring about change 
that is intended to result in improved perfonnance or outcomes or reduce the potential for 
noncompliance. As with Corrective Action and Enforcement, Improvement Plan Implementation 
leads to Reassessment to determine the impact of the planned activities. 

Two key characteristics pertain to Improvement Plan Implementation: 

• Gl. Plans and Resources for Implementation : Resources and support are provided for 
implementation of the improvement plan. 

• G2. Fidelity of Implementation : Plans are fully and evenly implemented in LEAs/EIS 
programs. 

Component: Reassessment 

The Reassessment component of the framework involves checking to see whether corrective actions 
or improvements have been effective following the implementation of a CAP or an improvement 
plan (including whether the LEA/early intervention agency is now meeting original targets or 
demonstrating progress). While Reassessment, like Problem Identification, includes comparisons 
between perfonnance and targets to gauge progress, Reassessment goes beyond this to also include 
the achievement of improvement plan or CAP goals and objectives. 

Three key characteristics pertain to Reassessment: 

• HI. Defensible Findings : The findings drawn from the reassessment process are 
adequately defended and supported by comparing LEA/EIS program perfonnance to 
specified targets. 

• H2. Complete Reporting : Reassessment reports completely describe the reassessment 
process so that essential infonnation is provided and easily accessible to LEAs/EIS 
programs and other stakeholders. 
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• H3. Dissemination of Reassessment Reports to Stakeholders : Reassessment reports are 

accessible and available to stakeholders. 



Summary of Study Objectives 

Before turning to the study methods and findings, it is useful to describe the study objectives in 
more detail. 

1 . Provide a description of the nature and scope of states’ monitoring systems. This 
objective was aimed at developing a description of what states did to meet their 
monitoring responsibilities under IDEA in 2004-05 and 2006-07. To meet this 
objective, the project team collected data on state monitoring systems and monitoring 
approaches and practices. The team also collected information on unique state 
circumstances that may influence state monitoring; for example, some states must 
monitor certain programs or indicators or use certain monitoring procedures based on 
a court order. 

2. Describe states’ monitoring systems at two points in time. The study included two 
site visits 2 years apart to Part B and Part C programs in each of 20 states. States’ Part 
B and Part C monitoring systems can be influenced by a variety of factors, such as 
changes to federal or state law, guidance from OSEP, technical assistance received 
from various sources (e.g., NCSEAM), and contextual factors (e.g., state size). These 
factors can in turn shape how monitoring is carried out by states in any given 
monitoring cycle. Therefore, it was believed to be important to collect data from 
different time periods in order to fully capture the variation among and within states’ 
monitoring systems under a range of contextual and other factors. Because the 
framework used to organize descriptions of state monitoring systems is preliminary, 
analyses of change over time or trend data are premature at this point. Consistent with 
this view, no statistics comparing the first and second rounds of findings are 
presented. 2 Instead, observed differences simply add another dimension to 
understanding of the variability of Part B and Part C monitoring systems in the 20 
states. 

3. Create a framework to describe state monitoring systems. This objective focused 
on developing and applying a framework that would provide OSEP and other 
stakeholders a greater understanding of the variation in the nature and design of 
states’ Part B and Part C monitoring systems. In order to develop this framework, 
project staff reviewed the extant literature on general standards for evaluation practice 
with input from experts on important features of a state monitoring system. The 
purpose of this study was not to evaluate or validate the framework itself. The reader 
should not infer that states that more closely emulate the framework have better 
monitoring systems or better outcomes. Additionally, the reader should recognize that 
the framework is not the only one that could be developed to represent state 
monitoring systems. 



2 See chapter 4 for further discussion on this issue. 
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4. Methodology 



The study included two data collections efforts: (1) a mail survey and (2) site visits. The mail 
survey data collection was conducted prior to the site visits and systematically gathered data on 
all states’ monitoring systems in 2004-05. The survey prepared the project team for the site visits 
by providing crucial preliminary insight into states’ monitoring activities, pointing to areas 
where survey information was insufficient to fully understand the intricacies of states’ 
monitoring activities and suggesting the kinds of on-site, in-depth interviews that would be 
needed to fully comprehend states’ monitoring systems. The mail survey data collection, 
including its methodology and its numerous findings, were discussed at length in an earlier 
report (Garrison-Mogren et al. 2008). Because the site visit data collection focused on a sample 
of states and used a very different methodology from the mail survey data collection, the findings 
from the two data collections are not directly comparable. 3 Therefore, the focus of this report is 
on the site visits. This chapter focuses on the site visit methodology, including the sampling 
methods used to select states, the steps taken to operationalize the framework, the procedures 
used to train site visitors, the site visit data collection process, the methods used to assess the 
reliability of the site visit data collection, and the scaling and scoring of the site visit data. 



Site Visit Data Collection 

The project team conducted two rounds of site visits with a random sample of 20 states to gather 
detailed information about the nature and design of these states’ monitoring systems. The first 
round of site visits took place in 2006 and focused on the states’ 2004-05 monitoring systems. In 
2008, the same 20 states were visited a second time to collect infonnation on each state’s 2006- 
07 monitoring systems. 4 The two rounds of site visits were conducted 2 years apart in order to 
capture the variation in the nature and design of states’ Part B and Part C monitoring systems 
under a range of contextual and background conditions. Analyses were not conducted to 
determine whether there was change over time. Such analyses were considered to be premature 
because the framework that was developed was intended to guide data collection and analysis 
and has not yet been validated or evaluated to determine if its adoption results in an improved 
monitoring system or better outcomes for children with disabilities and their families. 



3 The mail survey asked state agency staff questions about their state’s overall Part B and Part C monitoring 
practices, but it was not designed to delve into the many specific details about states’ Part B and Part C 
monitoring systems and processes that were explored during the site visit data collection. The mail survey data 
collection also did not incorporate the framework, as the framework was developed after the mail survey data 
collection. Therefore, even though some of the same topics were covered by the two data collections, the wording 
of the questions and the level of detail elicited about these topics was different, and the findings are not directly 
comparable. 

4 For Part B, states were asked about their monitoring systems in place during the 2004-05 and 2006-07 school 
years. For Part C, states were asked about their monitoring systems in place during the last completed monitoring 
period that most closely corresponded to the 2004-05 and 2006-07 school years; because Part C focuses on infants 
and toddlers with disabilities and not school-aged children with disabilities, the Part C monitoring cycles might 
correspond with the state’s fiscal year or the calendar year, as opposed to the school year. 
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The data collection procedures for the site visits included semi-structured interviews with key 
state and local infonnants and the collection and review of supporting documentation about the 
states’ monitoring systems. The information collected from the interviews and supporting 
documentation was then used in two ways: 

• First, because some way of condensing the vast amount of information that was 
collected during the site visits was needed, the project team created individual state 
profiles. 5 These state profiles included contextual issues within states that might have 
influenced their monitoring systems (e.g., state legislation, consent decrees, court 
cases, administrative issues), a descriptive narrative of the state’s monitoring system, 
and a discussion of how the state’s monitoring system mapped onto the framework 
and its components. 

• Second, this information was used to detennine whether the elements associated with 
the key characteristics and components of the framework were present or absent in 
each state’s monitoring system. 

Sampling Methods 

As noted in chapter 2, states can vary greatly in their approaches to the monitoring requirement. 
Because visiting all 50 states and the District of Columbia was cost prohibitive, the project team 
selected a systematic random sample of 20 states to capture the expected variability in states’ 
monitoring system designs. 

The process for selecting the 20 states to be included in the site visit data collection involved 
arranging states by characteristics believed to affect variability in monitoring and improvement 
practices and then using systematic random sampling to select states. The project team, with 
input from the Advisory Panel, selected (1) the number of children served by Part B and Part C 
of IDEA, (2) Part C lead agency, and (3) geographical region as the key state variables that 
should be used in the sampling process. For the number of children served, the project team used 
a combined special education and early intervention count of children served by Part B and Part 
C of IDEA in 2002-03. 6 For the Part C lead agency variable, three categories were used: 
education, health, and other (i.e., some other type of human services or social welfare agency). 
For geographic region, the project team used four Census Bureau regions to group states 
according to whether they were located in the North, Midwest, South, or West. 

In the first step, states were ordered according to the size of their combined special education and 
early intervention counts of children served. Next, states were divided into quartiles and then, 
within quartile, sorted by Part C lead agency and by region. The final step was to use systematic 
random sampling to select the 20 states to be included in the site visit data collection. 



5 Project staff used the individual state profiles as a means of organizing and standardizing the large amount of 
descriptive data that were collected during the site visits. Because the individual state profiles contained very 
specific state-level information that would reveal the identities of both the states selected for the site visits, as well 
as the key informants who were interviewed during the site visits, they are not included in this report. Chapter 5 
presents the information from the individual state profiles in aggregate. 

6 At the time that the sampling variables were chosen, the 2002-03 data were the most current child count data 
available; the child count is a census of the IDEA service population taken on a particular day each year. 
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Table 1 shows the distribution on the three sampling variables (i.e., Part B and C child count, 
Part C lead agency, and geographic region) for all 50 states combined, plus the District of 
Columbia, and for 20 sampled states. With regard to Part B and Part C child count, all states 
combined and the sampled states were evenly distributed among child count size groupings. For 
all the states combined and the sampled states, health was the most common Part C lead agency 
category (26 states and 9 states, respectively), followed by education (13 and 6 states, 
respectively), and other (12 and 5 states, respectively). All states combined and the sampled 
states were distributed similarly by region, with the South being the largest category for both (17 
states and 6 states, respectively), and the North being the smallest category for both (9 states and 
4 states, respectively). 

It should be noted that, although systematic random sampling was used to select states, the 
findings from the study focus on the sample of 20 states rather than attempting to generalize 
them to all states. In addition to the number of children served by Part B and Part C of IDEA, 
Part C lead agency, and geographical region, states vary on numerous other variables (e.g., their 
educational policies, state regulations, political organization, physical size, demographic make- 
up) that make it extremely difficult to generalize findings from this study to all states. 

Table 1. Distribution of all 50 states plus the District of Columbia and the 20 sampled 
states, by the three selection criteria: 2002-03 



Stratification variable 


All states 
(N= 51) 


Sampled states 
(N= 20) 


Child count 


Less than 40,000 


13 


5 


40,001 to 95,000 


13 


5 


95,001 to 175,000 


13 


5 


More than 175,000 


12 


5 


Part C lead agency 


Education 


13 


6 


Health 


26 


9 


Other 


12 


5 


Region 


Midwest 


12 


5 


North 


9 


4 


South 


17 


6 


West 


13 


5 



NOTE: The “all states” column includes data for the 50 states plus the District of Columbia. 

SOURCES: Data Accountability Center: Individuals with Disabilities Act (IDEA) Data. Retrieved March 2, 2005, 
from https://www.ideadata. 0 rg/arc_toc 6 .asp#partcCC; IDEA Infant and Toddlers Coordinators Association. 
Retrieved March 2, 2005, from http://www.ideainfanttoddler.org; U.S. Department of Commerce, Bureau of the 
Census. Retrieved March 10, 2005, from http://www.census.gov/geo/www/us_regdiv.pdf 

Steps to Operationalize the Framework 

Once the framework was finalized, the project team took steps to operationalize the framework 
so it could be used during the site visit data collection. These steps included converting the 
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elements from the framework into site visit rating elements, drafting element rating instructions, 
and creating a rating instrument for site visitors to use to record their ratings. 

Site Visit Rating Elements 

Each of the framework components included key characteristics and elements that described 
expectations for that component. For each element, the project team assigned either a 2-point or 
a 3 -point rating scale for site visitors to use to rate whether the element was present or absent in 
the state’s monitoring system. It should be noted that the site visit rating focused primarily on the 
presence or absence of particular elements in states’ monitoring systems as opposed to the 
“quality” of those elements; that is, “Was the element present?” as opposed to “How good is the 
element?” 

The primary scale was a 2-point scale (YES/NO). This scale was applied when there was a single 
instance of the element being rated. A “YES” rating affirmed that the element was present in the 
state’s monitoring system. A “NO” rating indicated that the element was absent. For example: 

Component: Problem Identification 

Key characteristic: Stakeholders are informed and contribute throughout the indicator 

and target setting process. 

Element: Parents of children with disabilities or representatives from 

advocacy groups represent the diversity that exists within the state. 
[YES/NO] 

In this example, the element is concerned with diversity in stakeholder participation in the 
indicator and target setting process. It should be noted that the element being rated is whether 
diversity is represented, not the number of groups represented (e.g., Black, White, rural, urban). 
For this reason, the 2-point scale was used. 

The other scale that was used was a 3-point scale (0, 1, or 2). This scale applied when there were 
multiple instances of the element being rated. A rating of “2” signified that the element was 
present for all instances in the state’s monitoring activities. A rating of “1” indicated that the 
element was present for some instances. A rating of “0” indicated that the element was absent for 
all instances. For example: 

Component: Problem Identification 

Key characteristic: Indicators are clearly written, logical, quantifiable, and represent 

valid measures of IDEA implementation. 

Element: Indicator definitions are clear and unambiguous. [0/1/2] 

In this example, the element concerns the existence of clear and unambiguous indicator 
definitions. Because states always had more than one indicator, the scale needed to reflect that 
there were multiple instances of this element being rated. That is, the element may be present for 
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ALL of a state’s indicators (a rating of “2”), SOME of a state’s indicators (a rating of “1”), or 
NONE of a state’s indicators (a rating of “0”). 

In some cases, a specific element may not have been relevant to a state’s monitoring system. For 
these situations, a rating of “NA ” (not applicable) was used. For example, if a state did not 
collect qualitative data during problem investigations, then site visitors used the “NA” rating for 
any elements that related to qualitative data analyses. In these instances, a rating of “NA” was 
used instead of a rating of “NO or NONE” because, if the state did not collect qualitative data, 
then it would not be expected to analyze qualitative data as part of its problem investigations. 

Element Rating Instructions 

In order to assist site visitors in making their ratings, the project team developed element rating 
instructions (see appendix C) and organized them by the framework components. Site visitors 
were provided with general guidelines for rating the element under each key characteristic, 
including a brief definition of the characteristic, the kinds of documentation that might be 
available for review (e.g., a list of the state’s indicators and targets, data collection procedures), 
the key infonnants the site visitors might need to interview, and whether the site visitors needed 
to see specific types of documentation in order to make ratings. Then, for each of the elements, 
the team provided site visitors with specific rating instructions, including more information about 
the element and the conditions under which they would rate an element as being present. 

Site Visit Rating Form 

The project team developed a site visit rating form so that site visitors could record their ratings 
in a standardized manner. This form organized the site visit rating elements by component and 
provided spaces for site visitors to make their ratings, along with any comments that they might 
have had for that element. 

Pilot Site Visits 

Before the first round of site visits, the project team conducted pilot site visits with two states 
that were not selected as part of the 20-state sample. In one state, the site visit focused on the 
state’s Part B monitoring system, and in the other state, the site visit focused on the state’s Part C 
monitoring system. Two different states were selected for the pilot study to reduce the burden on 
any individual state and to maximize variability. 

Teams of two senior staff members conducted each of the visits. For each pilot site visit, the 
teams conducted interviews and reviewed the monitoring documentation provided by the state. 
The objective was to detennine whether it was feasible by the end of the 2-day visit to describe 
the nature and design of the state’s monitoring system and to make ratings for each of the site 
visit elements using the element rating instructions and site visit rating form. The pilot site visits 
established that the planned timeframe was reasonable. However, the pilot site visits brought to 
light some issues and concerns about the wording and focus of particular rating elements. Using 
the pilot site visit experiences, project staff refined the rating elements and element rating 
instructions (wording changes, clarifications, etc.) and, in some cases, eliminated specific rating 
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elements altogether. This process resulted in a final set of rating elements that included 137 items 
(see appendix B for a complete list of the key characteristics and elements). 

The pilot she visits provided insight into three additional critical factors that necessitated 
refinements in the plans for the she visit data collection. The three factors were (1) 
inconsistencies in the tenninology being used by states to discuss monitoring, (2) the need to 
distinguish between state monitoring system designs and local-level monitoring efforts, and (3) 
the fact that states’ monitoring systems could be made up of multiple and distinct sets of 
activities. Each of these is discussed in turn below. 

Lack of Consistent Terminology 

It became apparent during the pilot site visits that states used common terms to mean very 
different things. As discussed earlier, the first data collection effort of this study involved a mail 
survey to all 50 states and the District of Columbia to find out more about states’ monitoring and 
improvement practices. Another purpose of the mail survey was to inform the site visit data 
collection. Therefore, the she visit teams carefully reviewed the mail surveys of the two states 
selected for the pilot she visits prior to conducting pilot site visits. 

While on-site, the teams gathered more infonnation and had the opportunity to gain further 
insight into states’ monitoring activities through comprehensive interviews. During the 
interviews with state agency staff, the pilot site visit teams determined that some common 
monitoring terms had state-specific meanings. For example, focused monitoring could mean that 
the state focused on a select number of LEAs or EIS programs each year, that it focused its 
efforts on a select number of monitoring priorities, or both of these (i.e., the state focused on a 
select number of monitoring priorities and on a select number of LEAs/EIS programs for each 
priority). Other examples of inconsistent usage included the term improvement plan, which could 
be used interchangeably with corrective action plan, and the terms validation and verification, 
which could also be used interchangeably. 

As a result of the pilot site visits, the project team decided that the site visitors should use the 
state’s tenninology when conducting interviews, rather than the language/terms used in the 
framework. However, when making ratings, site visitors were instructed to determine how the 
state’s tenninology mapped to the framework. For example, if the state used the term action plan 
when discussing how identified problems were addressed, site visitors were instructed to use the 
term action plan in their interviews, but, when making their ratings, site visitors were instructed 
to detennine if these activities mapped to Improvement Planning and Implementation, Corrective 
Action and Enforcement, or both of these components of the framework and then make their 
ratings accordingly. 

State Monitoring System Designs and Local-Level Monitoring Efforts 

During the pilot site visits, the project team determined that it also would be important for site 
visitors to be able to distinguish between state monitoring system designs and local-level 
monitoring efforts when making their site visit ratings. For example, according to the site visit 
rating elements, there should be stakeholder input into Problem Investigations. It might be the 
case that the state agency staff did not actually conduct the Problem Investigations; the 
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LEAs/EIS programs did. The project team was concerned that site visitors might feel that, in 
order to rate this element, they needed to determine whether every LEA/EIS program that 
conducted a Problem Investigation sought stakeholder input. 

To resolve this potential issue, the project team instructed site visitors that they would need to 
distinguish between state monitoring system designs and local-level monitoring efforts. So, in 
the example above, the site visitors would need to determine the intent of the state’s monitoring 
system design with regard to stakeholder input. If the state agency did not provide structure or 
direction to the LEAs/EIS programs for obtaining input from stakeholders during Problem 
Investigations, the site visitors would need to rate this element as being absent. That is, the 
state’s monitoring system design did not call for such input, and no procedures were in place for 
obtaining it. If, however, the state agency directed LEAs/EIS programs to include 
parent/advocate input during the Problem Investigation process and provided structure for them 
to be able to do so, and the site visitors confirmed this in follow-up interviews with one or two 
LEA/EIS program staff, then the site visitors would need to rate this element as being present 
because it was part of the state’s monitoring system design. The important point was that site 
visitors needed to be instructed and trained to rate the state-driven monitoring procedures that 
guided monitoring efforts at the local level (i.e., the state’s monitoring system design) and not 
rate individual local-level efforts. 

State Monitoring Systems Composed of Multiple Processes 

When developing the framework, it was assumed that states’ monitoring systems would be 
composed of a single set of activities. That is, site visitors could go to states and ask them to 
describe their Part B or Part C monitoring system, and states would describe a single set of 
activities that they used to monitor their LEAs/EIS programs. Instead, during the pilot site visits, 
the project team discovered that states could conceptualize monitoring as several distinct sets of 
activities. For example, a state might describe one monitoring activity as a “self-assessment,” 
whereby each year all LEAs/EIS programs collected and analyzed data to detennine their 
performance in a select number of a priority areas and then developed action plans to address any 
areas where they did not meet the target set by the state agency. The same state might then 
describe another monitoring activity as a “compliance review,” whereby the state agency staff 
conducted on-site record reviews with a select number of LEA/EIS programs to identify areas of 
noncompliance and then provided the LEA/EIS programs with a list of corrective actions that it 
had to implement. For the remainder of this report, these distinct sets of activities are referred to 
as “monitoring processes;” that is, a state’s monitoring system may be composed of a single set 
of activities or a single monitoring process, or it may be composed of multiple sets of activities, 
or multiple monitoring processes. 

The concept of multiple monitoring processes could complicate the site visit data collection. For 
example, the site visitors might view the “self-assessment” process described above as involving 
all five components of the framework — Problem Identification, Problem Investigation, 
Corrective Action and Enforcement, Improvement Planning and Implementation, and 
Reassessment. The “compliance review” process described above, however, could be interpreted 
as involving only two components — Problem Identification and Corrective Action and 
Enforcement. Under these circumstances, the site visitors would be faced with the dilemma of 
deciding how to describe the state’s two-process monitoring system overall and how to make 
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ratings for the components that occurred in two distinctly different processes (e.g., should they 
average the ratings somehow, and, if so, what would an average rating really mean?). 

For this reason, the project team decided to allow the state agency staff to explain the individual 
processes that composed their state monitoring system and then have the site visitors rate each of 
the processes separately. 7 That meant that the site visitors needed to agree on the number of 
distinct processes that made up a state’s monitoring system, gather enough infonnation via the 
interviews and documentation reviews to be able to fully describe each process and how it 
mapped onto the framework, and then be able to rate each process on each of the site visit rating 
elements. 

Training Site Visitors 

This section describes the procedures that were used to train site visitors and ensure the 
effectiveness of the training. 

Development of a Training Manual 

Prior to the training, project staff developed a site visit training manual. The manual contained 
information such as overviews of monitoring and IDEA, the study and its objectives, and the 
framework; a description of the site visit data collection procedures; item-by-item instructions 
for rating the site visit rating elements; a discussion of data sources for the site visit data 
collection; and the logistical procedures for scheduling the site visits. Trainees received the 
manual in advance and were instructed to review the materials ahead of time and be prepared to 
ask questions at the training. 

Training for First Round of Site Visits 



In March 2006, the project team held a 2-day training session for all project staff who would be 
site visitors for the first round of site visits. Overall, 18 individuals were trained; 17 actually 
conducted site visits. Each of these individuals had considerable research experience in 
conducting interviews and/or site visits; at minimum, site visitors were required to have a 
master’s degree in education or a related field. 

Trainees were required to attend the entire 2-day session. On the first day of the training, trainees 
received an in-depth presentation about the study to provide context for the site visit data 
collection. The presentation included a discussion of the framework, including a review of each 
of the components of the framework and the rating elements associated with them. Later in the 
day, trainees participated in a rating exercise using “mock interviews” created from information 
collected during the first pilot site visit. The project team audio-recorded these mock interviews. 
Trainees listened to the interviews as a group, with the recording being paused periodically to 
allow for group discussion of how they could use the information to make ratings and what 
additional information they would still need to collect. Trainees then made ratings, compared 
them to those made by project staff in advance of the training, and then discussed any 
discrepancies. 



7 This report presents findings for all state Part B and Part C monitoring processes; however, the need to protect the 
confidentiality of states precluded reporting some findings for state Part B and Part C monitoring systems. 
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On the second day of the training, trainees participated in a second rating exercise using 
information from the second pilot visit. Unlike the rating exercise on the first day, trainees 
worked independently (i.e., listening to the recorded interviews and reviewing documentation) to 
complete a set of site visit ratings. 

Agreement with a Standard Set of Ratings 

After the training, the project team compared trainee ratings from the second exercise to the 
ratings made by project staff before the training to identify areas that were still unclear to 
trainees. During the month following the in-person training, two additional conference calls were 
held with the trainees to further clarify areas that were identified as potential issues from the 
ratings in the second exercise, which included component definitions, rating instructions, and 
specific rating elements. 

During this same month, two senior project staff members who were well-versed in the site visit 
data collection methodologies conducted the first actual site visit. The materials from this site 
visit, including the state documentation and audio-taped interviews with state agency staff and 
other key informants, were used for the final training activity. For this activity, trainees reviewed 
the site visit materials and then made ratings. Trainee ratings were compared individually to a set 
of “standard” ratings compiled by two project staff members who reviewed the same site visit 
materials. (The original site visit team did not participate in the development of this standard set 
of ratings.) Trainees were required to have at least 80 percent agreement with the standard set of 
ratings before they could participate in the site visit data collection effort. 

Training for Second Round of Site Visits 

Fourteen of the site visitors from the first round of visits were available to conduct site visits 
during the second round of data collection. These returning site visitors participated in refresher 
training in advance of conducting any site visits. In addition, eight more staff were recruited to 
serve as site visitors for the second round. As with the other site visitors, these individuals had 
considerable research experience in conducting interviews and site visits. These new site visitors 
were trained according to the procedures described above, including the requirement to 
participate in a 2-day training in March 2008 and demonstrate at least 80 percent agreement with 
a standard set of ratings before being pennitted to conduct any site visits. A total of 22 
individuals participated in the second round of data collection. 

Data Collection Process 

The first round of site visits took place in 2006 and gathered information about state monitoring 
systems in 2004-05. The second round of site visits occurred in 2008 and gathered information 
about states’ 2006-07 monitoring systems. The various aspects of the site visit data collection 
process are described in more detail below, including notifying the sampled states, staffing the 
site visits, reviewing state documentation prior to the visits, conducting interviews and reviewing 
documents on-site, collecting follow-up information, finalizing ratings, and drafting reports. 
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Notifying Selected States 



In March 2006 and then again in February 2008, the project team sent introductory letters to the 
Part B state directors and Part C coordinators in the 20 selected states. The letters infonned each 
state special education director and Part C coordinator that their state had been selected as part of 
the site visit sample. The letter gave an overview of the study and site visit data collection 
procedures and provided contact information if states wished to find out more about the study. 

Staffing the Site Visits 

Project staff followed up with the Part B state directors and Part C coordinators to make sure 
they had received the introductory letter, obtain potential dates for the site visit, and identify a 
state contact who would handle site visit logistics. If there were any substantive questions about 
the study or site visit that needed to be addressed prior to arranging a site visit, a more senior 
project staff member called the Part B state director or Part C coordinator to discuss the 
question(s). 

A team of two site visitors conducted each site visit. On occasion, Part B and Part C visits were 
conducted together when the lead agency for Part C was the state department of education or 
when the Part B and Part C agencies were available for visits in the same week. However, the 
vast majority of site visits involved separate teams and separate dates for the Part B and Part C 
monitoring systems. Each Part B and Part C site visit was scheduled for 2 days; however, when 
the state department of education had oversight of both Part B and Part C, the site visits were 
combined and consolidated into 3 days when possible. 

After being assigned to a particular state, the site visitors made the detailed arrangements for the 
site visit themselves, communicating with the state contact about their travel plans and 
scheduling of interviewees on-site. 

Reviewing Pre-Site Visit Documentation 



The project team requested that states send documentation about their monitoring systems in 
advance of the site visits, so that the site visitors could review this information ahead of time, 
thereby minimizing the interview burden on the state agency staff. To facilitate this review, a 
documentation checklist was sent to the state contact (see appendix D). Requested 
documentation included general policies and procedures related to monitoring, a list of 
stakeholders that participated in any aspect of monitoring, a list of the indicators and targets used 
in monitoring LEAs/EIS programs, procedures for selecting LEAs/EIS programs for monitoring 
activities, descriptions of monitoring activities (e.g., on-site visits, self-assessments, etc.), and 
templates of reports related to monitoring findings. 

All documentation was sent to one project staff member, who logged it and sent copies to each 
assigned site visitor for review. 

Conducting the Site Visits 

The team then conducted the actual site visit. The site visitors needed to gather enough 
information during the site visit to be able to complete: (1) an internal site visit report for that 



30 




state that described the nature and design of the state’s monitoring system, the framework 
components that each of the state’s monitoring processes included, and how the state approached 
those components and (2) a set of site visit ratings that included ratings on all elements for each 
of the state’s monitoring processes. 

The site visits primarily consisted of interviews with state agency staff, representatives of local 
LEAs/EIS programs, and other stakeholders such as parents/advocates and members of state- 
level stakeholder groups, as well as additional documentation reviews. It should be noted that the 
site visit interviews collected retrospective infonnation (i.e., the site visits in 2006 collected 
information about states’ monitoring systems in place in 2004-05, and the site visits in 2008 
collected infonnation about states’ monitoring systems in place in 2006-07) and therefore relied 
upon the ability of key informants to recall the monitoring activities that took place in the 
previous year. As a result, the possibility that some interviewees may have inaccurately reported 
policies and procedures for the study’s two data collection periods cannot be ruled out. All 
interviews were audio-taped, with the permission of the interviewee, to enable the site team to 
check their notes against an audio record when necessary and for reliability purposes. Interviews 
with LEA/EIS program staff and other stakeholders were conducted privately, without state staff 
present, and often by telephone; this enabled the site visit team to limit their own travel time and 
include informants who were not within easy driving distance of state headquarters. 

As much as possible, site visitors used the first day of the site visit to gather in-depth information 
about the state’s monitoring system, particularly from the state agency staff. They began by 
interviewing the Part B state director or the Part C coordinator and the person in the state with 
the most knowledge of the monitoring process (in some states, the state director/coordinator 
served both roles). This initial interview provided an opportunity for the site visit team to furnish 
the state agency staff with more in-depth infonnation about the site visit, answer any questions 
they may have had, obtain an overview of the state’s monitoring system, and detennine what 
additional interviews they would need to conduct. 

A site visit interview guide (see appendix E) assisted site visitors in conducting these interviews. 
This guide included separate interview probes for the different types of interviewees, including 
state agency staff, local LEA/EIS program staff, and other stakeholders. The interview guide was 
not intended to be an exhaustive list of all of the questions, nor was it intended to serve as a 
questionnaire. It was intended to help site visitors begin discussions about the various topics and 
cover the essential topic areas, with the understanding that the site visitors could reorder, adapt, 
and ask additional questions depending on the level of detail provided by the interviewees. 

Throughout the first day, site visitors continued interviews with state agency staff. Between 
interviews, the site visitors conferred with one another about the information that they had heard 
during the interviews, referred to the documentation to confirm that the information was 
consistent with the materials that had been provided, and planned for their next interviews. At 
times, it was necessary for the site visit team to request additional documentation from the state 
agency staff to clarify or support infonnation from the interviews. 

At the end of the first day, site visitors scheduled time to discuss their preliminary 
conceptualizations of the state’s monitoring system with each other. They discussed issues such 
as: How many monitoring processes made up the state’s monitoring system? What were those 



31 




processes and how would they describe them? What components from the framework were 
present in each process? What would the monitoring processes look like if they sketched them 
out? How did the processes map onto the framework and its components? Site visitors were also 
instructed to independently make as many initial ratings as they could with regard to the site visit 
rating elements and then to discuss those ratings to see where the two team members agreed, 
disagreed, or felt they needed more information. Site visitors were also instructed to plan the 
second day by identifying what further information was needed to complete their ratings (e.g., 
clarification, documentation) and their internal site visit reports. 

Site visitors then used the second day of the site visit to fill in any gaps that they identified, using 
the same procedures as before (i.e., conducting interviews and follow-up interviews, referring to 
the documentation, and conferring with each other as needed). In addition, the site visitors 
conducted interviews with the local LEA/EIS program staff and stakeholders to corroborate the 
information that they had received from the state agency staff and to gather additional 
information about the state’s monitoring system. The site visitors also tried to finalize as many of 
their ratings as possible. This typically involved revisiting any rating elements on which they 
initially disagreed or needed more information and seeing if they could now come to agreement. 

Collecting Follow-Up Information 

Sometimes it was necessary for the site visit team to collect follow-up information after leaving 
the state. For example, site visitors occasionally could not complete telephone interviews while 
on-site because they could not reach a particular individual, often a stakeholder or LEA/EIS 
program staff member. The site visit team completed these interviews at a later date. At other 
times, the site visitors requested additional documentation or sent follow-up questions to the state 
by e-mail if they discovered they still lacked a key piece of information to make their ratings. All 
site visitors found a need to engage in these types of follow-up activities. 

Post-Site Visit Activities 



Upon completing the site visit, each site visit team needed to finalize its ratings of the site visit 
elements and draft an internal site visit report that summarized the descriptive information that 
had been gathered during the course of the site visit. 

Finalizing Site Visit Ratings 

Ideally, the site visitors finalized their ratings of the site visit elements before leaving the state. 
This typically involved the team members going over each element and discussing any elements 
for which there was no immediate agreement, until the two team members came to an agreed- 
upon rating for each element. If the site visit team members could not finalize their ratings on- 
site due to the need to collect additional information, they completed any necessary follow-up 
activities, conferred, and then submitted their final set of ratings within 2 weeks of the site visit. 
It should be noted that, although site visitors were instructed to independently come up with an 
initial set of ratings, they were not asked to submit these independent ratings because of the 
iterative nature of the rating process, which required site visitors to discuss their ratings 
throughout the 2-day visit to determine elements on which there was disagreement or the need 
for additional infonnation and then revise their ratings accordingly. 
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Drafting Internal Site Visit Reports 



After the site visit was completed, the site visit teams were also required to draft an internal site 
visit report that would later form the basis of the state profile for that state. The project team 
provided site visitors with an internal site visit report template in order to standardize the type of 
information included in the report, as well as the level of detail provided. A senior project staff 
member who did not conduct the visit reviewed each site visit report to ensure the consistency 
and clarity of the information provided. If any questions or issues arose, project staff 
communicated with the site visit team to resolve them and to revise the report. 



Development of Individual State Profiles 

Once the internal site visit reports were finalized, project staff then transformed the reports into 
individual state profiles by taking the responses that site visitors provided to each of the 
questions and turning them into narrative descriptions of the state’s Part B and Part C monitoring 
systems. The individual state profiles provided information on the following topics: 

• contextual factors, including the number of LEAs/EIS programs; resources devoted to 
monitoring (e.g., staffing, whether any consultants or subcontractors were used, if the 
state worked with NCSEAM, whether the state had a management information 
system that was used for monitoring); any state legislation, legal issues (e.g., court 
cases, consent decrees, compliance agreements with OSEP), or administrative issues 
that may have affected the state’s monitoring system (e.g., recent changes in lead 
agency or key staff, staffing issues, significant reorganizations); and if it was a year of 
transition in terms of the state’s monitoring system (e.g., a new monitoring process 
being implemented, an old monitoring process being phased out); 

• the design of the state’s monitoring system, including how many monitoring 
processes the system included, the names of those processes, and how those processes 
were carried out (e.g., who was responsible for implementing the process, how often 
the process was initiated, how LEAs/EIS programs were selected for the monitoring 
process, and the key components/phases/procedures of the process); and 

• how the state’s monitoring processes mapped onto the framework and its 
components, including which components were present for each process and the 
procedures that the state had in place for carrying out that particular component (i.e., 
if a process included Problem Investigation, who was involved and how did the state 
go about doing this?). 

It should be noted that state profiles contained the same level of information as the internal site 
visit reports, but the information was presented in a more standardized manner in order to 
facilitate the coding and analysis of this information. To code this infonnation, project staff first 
reviewed the individual state profiles and identified key variables that described the nature and 
design of states’ Part B and Part C monitoring systems in 2004-05 and 2006-07. Two project 
staff members then reviewed the state profiles and independently coded the available information 
for each of these variables for each of the Part B and Part C monitoring systems and processes. 
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After comparing their initial coding results, the two project staff members discussed and resolved 
any discrepancies. The resulting data were then aggregated so that findings could be presented 
for the Part B and Part C monitoring systems and processes for both 2004-05 and 2006-07. 



Reliability of Site Visit Data Collection 

A reliability study was conducted to determine if there was consistency across site visit teams in 
their ratings of states’ Part B and Part C monitoring processes (see appendix F for more 
information). The reliability study involved sampling the completed site visits from eight states. 
Randomly selected two-person comparison teams then developed new ratings for those eight site 
visits. The comparison teams did not make actual site visits. Instead, they used audio recordings 
of the original site visits and available documentation to complete their ratings. The ratings of the 
original site visit team and the comparison team were compared to establish an “inter-team” 
reliability. The data were aggregated to give an overall score for each set of raters across the five 
framework components. Then, an intraclass correlation coefficient (ICC), which is a measure of 
inter-rater reliability, was calculated for the total score. The result for the 2004-05 reliability 
study data was 0.84. The result for the 2006-07 reliability study data was 0.77. Acceptable levels 
of reliability are typically considered to be values greater than 0.70 (see Stemler 2004). 



Development of Component Scores 

To determine the extent to which the elements associated with each component of the framework 
were present in states’ Part B and Part C monitoring processes, the ratings for the individual 
elements that made up each component were combined into an overall score for that component. 
This task required two steps: (1) reducing the set of elements for each component of the 
framework to an efficient and reliable measure and (2) creating component scores using Item 
Response Theory (IRT) modeling. 

Reduced Set of Rating Elements 

In the first step, the project team explored whether the elements associated with each component 
of the framework could be reduced to a smaller number of items. During the first round of site 
visits, site visitors rated 137 elements — a process that was quite time-consuming for them and 
potentially burdensome to states (i.e., to the extent that rating all of these elements required extra 
time with state officials). If some of the elements did not contribute distinctive information or 
were duplicative, it made sense to eliminate them for the second round of visits and work with a 
more manageable and efficient set of rating elements. 

To examine this possibility, the project team used a statistical technique called Rasch analysis, 
which applies an objective set of rules for considering how much each individual element 
contributes to an overall score. Appendix G includes a brief introduction to the Rasch procedure 
and describes in more detail how it was applied to the first round of site visit data. The goal was 
to find the most efficient (smallest) set of elements possible for each component of the 
framework. The analysis helped to achieve this goal by reducing the total number of elements 
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across the five components from 137 to 65 while still maintaining high levels of reliability for 
each component. 

Creation of Component Scores 

In the second step, the project team calculated scores for each of the framework components in 
order to determine the extent to which the elements associated with each component were 
present in states’ Part B and Part C monitoring processes. One approach that the project team 
considered using was just to calculate simple percentages by dividing the number of elements 
present for each Part B or Part C monitoring process for a given component by the total number 
of elements for that component. However, instead of using this approach, IRT modeling was 
used to create the component scores for two primary reasons. First, IRT took into consideration 
not only which elements were rated as present or absent, but also the frequency patterns of the 
elements. This means that, when the scores were created, some elements were given more weight 
than other elements because of their specific properties (i.e., the frequency with which particular 
elements were rated as present or absent). Second, IRT was used because the second round of site 
visits used a shorter instrument than the first (i.e., 65 elements, rather than 137 elements). Using 
IRT modeling allowed the project team to apply a linking method (Stocking and Lord 1983) to 
the 65 elements that were used in both rounds of site visits to create a single metric. This was 
done so that the scores from the first round of site visits would be comparable to scores from the 
second round of site visits. 

The final product of these analyses was two sets of scores for each component of the framework 
(i.e., one for each round of site visits) for each state monitoring process. These scores are 
referred to as component scores in subsequent chapters. The results of the IRT modeling were 
transformed so that each component score represents the estimated percentage of the elements 
for a particular component that were present for an individual monitoring process. So, for 
example, if a process received a component score of 58 for the Problem Investigation 
component, it means that an estimated 58 percent of the elements for this component were 
present for that process. More detailed information about the development of the component 
scores can be found in appendix H. 
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5. Study Findings 



The primary objective of this study was to describe the nature and scope of states’ monitoring 
systems. Two additional interrelated objectives were to describe states’ monitoring systems at 
two points in time and to create a framework to describe state monitoring systems. To address 
these objectives, the project team conducted site visits to a systematic random sample of 20 
states at two points in time, once in 2006 focusing on states’ Part B and Part C monitoring 
systems and processes in 2004-05 and then again in 2008 focusing on states’ Part B and Part C 
monitoring systems and processes in 2006-07. In preparation for the site visits, the team 
developed a framework for monitoring to guide the data collection, along with protocols for 
interviews and forms to rate whether the elements associated with the framework were present or 
absent in state monitoring systems and processes. Following the site visits, the project team 
generated state profiles for each state’s Part B and Part C monitoring systems based on 
interviews with key informants and documentation provided by the state. In addition, team 
members compiled the site visit ratings across all of the states’ Part B and Part C monitoring 
processes. 

This chapter presents findings from the analyses of states’ Part B and Part C monitoring systems 
and processes, including a systematic review of the descriptive information contained in the state 
profiles for each round of site visits and an analysis of the ratings of the elements associated with 
the framework. The chapter begins with an overview of the context for states’ Part B and Part C 
monitoring systems. The second section provides an overview of state approaches to monitoring 
Part B and Part C. Then the next section discusses the extent to which states’ Part B and Part C 
monitoring processes mapped onto the framework, first in general and then more specifically 
with regard to how states went about identifying, investigating, addressing, and reassessing 
problems. The final section presents an analysis of the ratings of states’ Part B and C monitoring 
processes for the components and the individual elements of the framework. 

Each section presents data from 2004-05 and then 2006-07. Information on the two time periods 
is intended to provide two windows on the variability in state monitoring and improvement 
practices in a sample of states. As noted in chapter 3, any differences between the two time 
periods should not necessarily be interpreted as changes over time or trends in these 20 states, 
nor should the differences be assumed to reflect changes at the national level across all states. In 
addition, it should be noted that the total number of Part B and Part C monitoring processes 
observed was greater than the number of states. The findings presented in this chapter primarily 
focus on states’ Part B and Part C monitoring processes as opposed to states’ Part B and Part C 
monitoring systems. Because there were only 20 Part B and Part C monitoring systems for each 
data collection period, these data often did not meet reporting standards due to small cell sizes 
that could potentially identify individual states. Where possible, data on both state monitoring 
systems and processes are presented, but, in many cases, only data on state monitoring processes 
can be presented. Analyses of state monitoring processes necessarily overrepresent states with 
multiple processes; however, in these instances, the focus is on the description of the monitoring 
processes, not the state monitoring systems. 
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The Context for State Monitoring 



Beyond federal mandates and guidance from OSEP, many factors may shape the way that states 
approach monitoring for Part B and Part C. States vary in size and density, are distributed across 
regions with different economic and other challenges, and have differing political and 
bureaucratic frameworks. These and other factors constitute the broad environment for IDEA 
monitoring and may affect a variety of state decisions, including what agency takes the lead for 
Part C, the level of resources available for monitoring, and how monitoring resources are 
deployed. 

The current study was not designed to explore the complex interactions among these contextual 
factors. However, the site visit protocols did require site visitors to collect some background 
information about the monitoring environment, including how states staffed their monitoring 
efforts and what organizational units were monitored. In addition, state agency staff were asked 
to describe any transitions underway in their monitoring approaches as well as any other features 
of the state environment that would help the site visit team understand the states’ monitoring 
systems in 2004-05 and 2006-07. This infonnation was incorporated into the site profiles that 
were developed from the first and second rounds of site visits. 

This section describes some of these contextual features, based on information extracted from the 
profiles. (The distribution of sampled states by region, Part C lead agency, and size of child 
count was described in chapter 4 — see table 1 .) 

The Context for Part B Monitoring 

This section looks first at the contextual issues for Part B monitoring systems, including the 
monitoring units involved; the resources deployed for monitoring; the status of state monitoring 
systems, including whether states were implementing any new monitoring processes; and other 
factors that states mentioned as being particularly influential in shaping their monitoring 
practices. 

Part B Monitoring Units 

As noted earlier, all the Part B monitoring systems were administered by the state's department 
of education (SEA). 8 All 20 states (100%) reported that the key monitoring unit was LEAs. In 
addition, states identified a variety of other types of monitoring units, including intennediate 
education agencies or cooperatives 9 (identified by 10, or 50% of all states), state-operated 
schools or programs (16 states, 80%), charter schools that were autonomous LEAs (7 states, 
35%), private schools (9 states, 45%), and other entities (8 states, 40%). The total number of 
monitoring units varied greatly within the 20 states. Six states (30%) had more than 500 



s In addition to ensuring that local programs are carrying out their responsibilities under IDEA, states have 
numerous administrative responsibilities under the law. For example, states must find children in the state who are 
eligible to receive special education services (child find); ensure that all children with disabilities, to the extent 
possible, are educated with children without disabilities; and distribute funds to local programs. Additional state 
responsibilities under IDEA are enumerated in Sections 611 and 612 of the act. 

9 An intermediate education agency is an education agency at the county or regional level that exists to provide 
specialized instructional and administrative support and services to the LEAs within its geographic area. 
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monitoring units; nine states (45%) had from 100 to 499; and five states (25%) had fewer than 
100 monitoring units. 

In some states, students in schools for the deaf or blind, as well as students in correctional 
facilities and other residential placements such as mental health centers, hospitals, etc., were not 
considered part of any monitoring unit, and the state had separate procedures for monitoring 
compliance with IDEA in these locations. The current study did not examine these separate 
procedures. 

Because LEAs constituted, on average, 75 percent of a state’s monitoring units, throughout the 
remainder of this report, the tenn “LEAs” is used as shorthand for “Part B monitoring units” — 
recognizing that the term is also meant to encompass intermediate monitoring units and any 
entities such as charter schools that were treated like LEAs or components of LEAs. 

Resources Used in Monitoring Part B Systems 10 

In both time periods, site visitors collected infonnation on the human resources that the 20 states 
devoted to monitoring — specifically, whether state staff carried out monitoring responsibilities 
themselves or with contractor/consultant assistance and whether there had been any staffing 
issues that affected monitoring. 

A number of states used contractors and/or consultants and stakeholders to assist state staff with 
various tasks. In 2004-05, 7 of the 20 states (35%) used consultants and/or contractors to 
complete various monitoring activities. In 2006-07, 12 states (60%) used contractors or 
consultants. Consultants/contractors were tasked with conducting on-site monitoring visits, 
providing technical assistance to LEAs on monitoring practices, and helping LEAs develop 
corrective action and improvement plans. Four states mentioned that staff shortages or staff 
turnover had been a concern in 2004-05 and had affected their ability to monitor. Four other 
states (not the same ones as in 2004-05) mentioned staffing issues in 2006-07. 

Site visitors also determined whether the state had worked with NCSEAM to develop its Part B 
monitoring system in 2004-05 and 2006-07. NCSEAM was an OSEP-funded project that began 
working with states in 2002 to assist them in developing or redesigning their monitoring systems. 
In 2004-05, a total of 4 of the 20 states (20%) indicated that they had worked with NCSEAM to 
develop one or more of the monitoring processes that made up their Part B monitoring system. In 
2006-07, 10 states (50%) had worked with NCSEAM. 

In 2006-07, site visitors also systematically documented whether states had a statewide 
electronic database available to assist them in their monitoring tasks. (Site visitors had collected 
some information relevant to this topic in 2004-05 as well, but they were not required to 
document it systematically in the profiles; therefore data for 2004-05 are not reported.) In 2006- 
07, all 20 states reported that they had a management information system (MIS) that was used 
for monitoring; some had more than one. States varied considerably in how they used such 
systems. At one end of the spectrum, the systems were used primarily to collect and report data 
from LEAs that would be needed to file federally mandated reports. At the other, MIS databases 

10 While it is common to think of resources in terms of money, it was beyond the scope of this study to conduct a 
financial analysis of state support for monitoring. 
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were used for a wide array of purposes — to assess LEAs’ compliance and/or performance on 
indicators, set indicator targets, select LEAs to be monitored, select student records to be 
reviewed, and verify district data. Depending on the design, databases might allow state officials 
to examine the reasons for noncompliance, track the correction of noncompliance by local 
programs, and detennine the impact of strategies designed to improve outcomes for children with 
disabilities. MIS systems varied in their procedures for data entry as well, from those that 
required LEAs to submit their data annually on discs for input at the state level to web-based 
systems that pennitted LEAs to input data online as often as they wished. None of the databases 
was used for individual case management by local LEAs. 

The Status of the States’ Part B Monitoring Systems 



In 2004-05 and 2006-07, a number of states were in transitional periods with regard to their Part 
B monitoring systems (9 states, or 45%, and 12 states, or 60%, respectively). Individual 
situations varied, ranging from states that were in the midst of developing or pilot-testing new 
monitoring processes and procedures while phasing out old ones to states that were fully 
implementing new monitoring processes for all LEAs. 

Other Factors Reported to Influence Part B Monitoring Systems 

In both 2004-05 and 2006-07, site visitors asked states about other legislative, legal, or 
administrative issues that might have influenced their Part B monitoring systems. The issues 
reported varied a great deal and were often unique to a particular system — for example, high 
levels of staff turnover in certain locations or a department reorganization that affected 
monitoring responsibilities. However, one type of issue was mentioned in 3 or more of the 20 
states. In 2004-05, 6 states (30%) reported that lawsuits, consent decrees, or a compliance 
agreement with OSEP had influenced their monitoring systems. 11 Depending on the particular 
instance, these legal issues affected the types of indicators used, the scope and timing of 
monitoring, or the deployment of monitoring staff. In 2006-07, five of these same states (25%) 
reported that legal issues continued to influence monitoring. 12 

The Context for Part C Monitoring 

This section describes the contextual issues for Part C monitoring systems, covering the same 
four categories discussed for Part B: the monitoring units used; the resources deployed for 
monitoring; the status of state monitoring systems, including whether states were implementing 
any new monitoring processes; and other factors. 



1 1 In this context, a consent decree refers to a judicial order that formalizes an agreement negotiated between a 
defendant (typically a school system or a state agency responsible for administering IDEA) and a plaintiff (often a 
parent or group of parents who have challenged the actions of the school system or state agency on behalf of their 
child or children). A compliance agreement, as provided under the General Education Provisions Act (GEPA) and 
referenced in IDEA, refers to an agreement between ED and a state agency stating the terms and conditions that 
the state must meet in order to be in full compliance with federal statute; if the state fails to comply with the terms 
and conditions, ED may take any action authorized by the law (20 USC 1234 §f). 

12 Note that the consent decrees cited were not necessarily still in effect. States could retain approaches developed as 
a result of such agreements after a decree was terminated. 
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Part C Monitoring Units 



As noted earlier, the lead agency for Part C was not always the department of education. The 
department of health was the lead agency in 9 of the 20 states (45%); another human service 
agency took the lead in 5 states (25%); and the department of education was lead agency in the 
remaining 6 states (30%). Because of the variety of lead agencies and the ages of children 
involved (0 to 3), EIS programs were organized in many different ways. States might use 
regional or local agencies, or some combination thereof, to manage and deliver Part C services. 
They might assign responsibility for intake and case assessment to one type of agency and 
responsibility for delivery of direct services such as family counseling, physical and speech 
therapy, home visits, and the like, to other agencies. They could opt to use existing local or 
regional structures that were already in place — such as a county or regional health department, a 
regional education agency (encompassing multiple school districts), or a private service agency. 
Agencies involved in managing Part C services also might be selected through a competitive 
process. In addition, Part C programs relied on additional contracted providers (individuals or 
local agencies) to meet the needs of their children and families. 

Because the approaches to service delivery were so diverse, the local monitoring units were 
equally so. The total number of monitoring units varied considerably across the 20 states; 4 states 
(20%) reported more than 100 monitoring units; 7 states (35%) reported 20 to 99 monitoring 
units; and 9 states (45%) reported fewer than 20 monitoring units. Throughout the remainder of 
this report, the term “EIS programs” will be used as shorthand for “Part C monitoring units” — 
recognizing that the tenn encompasses public and private agencies, as well as individual 
providers in some instances. 

Resources Used in Monitoring Part C Systems 

As with Part B monitoring systems, some states used contractors and/or consultants and 
stakeholders to assist state agency staff with monitoring EIS programs. Ten of the 20 states 
(50%) hired consultants and/or contractors in 2004-05; 12 states (60%) did so in 2006-07. Four 
states (20%) mentioned that staff shortages or turnover had been a concern in 2004-05 and had 
affected their ability to monitor. Seven states (35%) mentioned staffing issues in 2006-07. 

As with Part B, site visitors also determined whether the state had worked with NCSEAM to 
develop its monitoring system in 2004-05 and 2006-07. In both 2004-05 and 2006-07, four states 
(20%) indicated that they had worked with NCSEAM to develop one or more of the monitoring 
processes that made up their Part C monitoring system. 

In 2006-07, a number of the 20 states reported that they had an MIS that was used for 
monitoring; some had more than one. In general, the Part C databases were used in the same 
ways as were the Part B databases, as states reported using them to analyze EIS program data, set 
indicator targets, select EIS programs to be monitored, select child records to be reviewed, and 
verify EIS program data. However, unlike Part B, the Part C lead agencies generally did not use 
these databases to examine the reasons for noncompliance, correction of noncompliance by local 
programs, and the impact of strategies designed to improve outcomes for children with 



13 Exact numbers are not reported to prevent state monitoring systems from being identified. 
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disabilities. As with Part B, MIS systems varied in their procedures for data entry, from those 
that required EIS programs to submit their data periodically for input at the state level to web- 
based systems that permitted EIS programs to input data online as often as they wished. Three of 
the databases also were used for individual case management by local EIS programs. 

The Status of States’ Part C Monitoring Systems 

In 2004-05 and 2006-07, several states were in transitional periods with regard to their Part C 
monitoring systems (9 states, or 45%, and 7 states, or 35%, respectively). As with Part B, 
individual situations varied, ranging from states that were developing or pilot-testing new 
monitoring processes and procedures while phasing out old ones to states that were fully 
implementing new monitoring processes for all EIS programs. 

Other Factors Reported to Influence Part C Monitoring Systems 

In both 2004-05 and 2006-07, states identified a number of legislative, legal, and administrative 
issues as influencing their Part C monitoring systems. As with Part B, the issues cited by Part C 
monitoring systems varied a great deal and were often unique to a particular state, ranging from 
state legislation that affected certain monitoring activities to changes in a state’s Medicaid 
program. However, a few types of issues were mentioned in 3 or more of the 20 states. In 2004- 
05, three states reported that a consent decree or a compliance agreement with OSEP had 
influenced their monitoring systems. In 2004-05, four states cited administrative issues that had 
affected monitoring, involving either the complexities of their organizational structure or 
changes in responsibility for Part C monitoring. In 2006-07, four states reported that financial 
difficulties were affecting their programs; none of the states had cited these factors in the earlier 
time period. 

Summary for the Context for State Monitoring 

This section has reviewed some of the contextual factors that may affect Part B and Part C 
systems and processes. A number of the states used consultants or contractors to help carry out 
their monitoring responsibilities — including 7 of the 20 Part B monitoring systems (35%) and 10 
of the 20 Part C monitoring systems (50%) in 2004-05, and 12 of the 20 Part B and Part C 
monitoring systems (60%) in 2006-07. Four Part B monitoring systems in each time period 
(20%) mentioned staffing shortages or turnover as a concern, as did four Part C monitoring 
systems (20%) in 2004-05 and seven (35%) in 2006-07. A number of monitoring systems were 
in transition in each time period: this was the case for 9 Part B and Part C monitoring systems 
(45%) in 2004-05 and 12 Part B monitoring systems (60%) and 7 Part C (35%) monitoring 
systems in 2006-07. 

Part B and Part C officials cited many other contextual factors that affected their monitoring, but 
few types of issues were cited by more than one state. Issues related to lawsuits, consent decrees, 
or compliance agreements with OSEP were exceptions. For Part B, six states (30%) reported that 
these issues had influenced their monitoring systems in 2004-05, and five of these same states 
(25%) reported that these issues continued to influence monitoring in 2006-07. For Part C, three 
states reported that a consent decree or a compliance agreement with OSEP had influenced their 
monitoring systems. Other factors that were cited by three or more states in either time period 
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included administrative issues related to organizational structure or program responsibility (four 
Part C monitoring systems in 2004-05) and financial issues (four Part C monitoring systems, 
2006-07). 



Overview of Approaches to Monitoring 

As noted earlier, states may use one or more processes to pursue their monitoring objectives. 
Taken together, all of the state’s monitoring processes are defined as the state’s monitoring 
“system” in the terminology of this report. 

Later sections systematically examine how and whether monitoring processes and systems in the 
20 states incorporated the framework components. Before proceeding to that level of detail, 
however, this section provides an overview of approaches in these states — focusing on a few 
broad attributes that were observable across processes and systems, regardless of how states 
implemented the monitoring components. These attributes include the following: 

• whether state monitoring systems relied on single or multiple processes in carrying 
out their monitoring objectives; 

• whether monitoring systems and processes addressed all LEAs/EIS programs; 

• what entity was primarily responsible for implementing the monitoring; and 

• how often monitoring processes were initiated. 

Infonnation on the first attribute — whether a state was a single-process or multi-process 
system — was derived from the site visit rating instruments for the first and second rounds of 
visits. The primary source of data for the remaining three attributes was the comprehensive site 
profiles that were prepared after each round of site visits, which contained an overview of each 
state’s approaches and individual descriptions of each process a state used. 

Number of Monitoring Processes 

Table 2 displays the number of monitoring processes identified in each round of site visits for 
Part B and Part C. In 2004-05, the site visitors identified 58 distinct monitoring processes. Part B 
accounted for 34 monitoring processes, or an average of 1.6 processes per system, and Part C for 
24 monitoring processes, or an average of 1 .2 per system. In 2006-07, site visitors identified 60 
monitoring processes — 32 for Part B, or an average of 1.6 per system and 28 for Part C, or an 
average of 1 .4 per system. 

All states had at least one Part B and one Part C monitoring process. For Part B monitoring 
systems, 10 states (50%) implemented more than one process in 2004-05 and 2006-07. For 
Part C monitoring systems, three states (15%) implemented more than one process in 2004-05, 
and seven states (35%) reported more than one in 2006-07. 
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Table 2. 



Number of state monitoring processes and multi-process state monitoring 
systems identified, by part: 2004-05 and 2006-07 





Part B 


Part C 




monitoring systems 


monitoring systems 




(N=20) 


(N=20) 


Number 


2004-05 2006-07 


2004-05 2006-07 


Monitoring processes 


34 32 


24 28 


Systems with more than one process 


10 10 


3 7 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 



Monitoring Process Types 

To further examine approaches to monitoring, the project team developed a typology of 
monitoring processes. This typology was based on what the project team determined were two 
critical decisions that states must make when implementing monitoring systems — which 
monitoring units to review and who should collect, review, and interpret data on IDEA 
implementation. 

The first type of decision made by states pertains to which monitoring units are reviewed. States 
must decide whether the monitoring process should be applied to all LEAs/EIS programs or only 
to those that meet some specific performance criteria — criteria designed and used to identify low 
performers. The rationale for basing monitoring on performance appears in recent thinking about 
ways to allocate limited financial and human resources for monitoring (e.g., Comstock-Galagan 
and O’Connell 2002), as well as in comments by state agency staff during the site visit 
interviews. According to these sources, the argument for selecting programs for monitoring 
based on performance is that state resources are directed to LEAs/EIS programs that are deemed 
in most need of assistance or intervention. 14 The alternative to a monitoring process based on 
performance is one that is designed to reach everyone. In the latter approach, every LEA/EIS 
program is monitored on a set schedule regardless of performance — although not necessarily 
every year. To distinguish between these two approaches, the project team labeled the former 
“performance-based” assessment processes and the latter “schedule-based” assessment 
processes. 

The second type of decision made by states pertains to the entity primarily responsible for 
implementing the monitoring process — the state agency, the LEA/EIS program, or both. 
Although there is no research indicating which approach may be more effective, state agency 
staff during the site visit interviews reported two rationales for giving LEA/EIS programs 
substantial roles in the monitoring process. One rationale state officials reported was that local 
involvement promotes local ownership of identifying and resolving problems. According to this 
view, local personnel and other stakeholders who participate fully in their own monitoring 
(rather than having it “done to” them) will be more invested in improving practices and 



14 Because some LEAs/EIS programs might never be subject to this type of monitoring process if they always 
performed above the system’s threshold for attention, four systems supplemented their selections of low 
performing programs with others, such as randomly chosen LEAs/EIS programs or programs that volunteered for 
the process. 
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outcomes. The other rationale was that local involvement saves state resources by relieving state 
agency staff and their teams (or contractors) of some of the burden of data collection and 
problem identification. Savings are, of course, partially offset by additional state investments in 
training and tools designed to support local agencies and programs in their self-monitoring 
responsibilities. The rationale offered for the alternative — having a monitoring process carried 
out primarily by the state — is that it can enable more consistent application of monitoring 
procedures statewide. However, state resources may limit the number of LEAs or EIS programs 
that can be covered by that monitoring process in any given year. 

Based on these two types of key decisions that states needed to make for their monitoring 
systems, the processes identified in the state profiles were classified as to whether they were (1) 
performance based (i.e., applied selectively to low performers) or schedule based (i.e., applied to 
all monitoring units on a schedule) or (2) carried out primarily by the state, LEAs/EIS programs, 
or both. 15 

As a result of this exercise, the project team identified four types of processes: 

• processes that were perfonnance based and made the state primarily responsible for 
implementation were labeled performance-based state assessment; 

• processes that were schedule based and made the state primarily responsible for 
implementation were labeled schedule-based state assessment; 

• processes that were schedule based and made the LEA/EIS program primarily 
responsible for implementation were labeled schedule-based local assessment, and 

• processes that were schedule based and made the LEA/EIS program and state agency 
each responsible for certain phases of implementation were labeled schedule-based 
hybrid assessment. 

Table 3 below provides detailed definitions of the four observed process types. The only 
performance-based process is performance-based state assessment. Two other logical 
possibilities, the combination of performance-based process with LEA/EIS responsibility (i.e., 
performance-based local assessment) and the combination of performance-based process with 
shared LEA/EIS and state responsibility (i.e., performance-based hybrid assessment) were not 
observed in the 20 states. Regardless of the label, keep in mind that all processes are the ultimate 
responsibility of the state. 16 



15 The project team developed initial guidelines for classifying processes, and then two staff members independently 
categorized each process. Additional criteria were introduced when needed to resolve ambiguity in the original 
definitions. The final criteria are reflected in the definitions shown in table 3. 

16 While performance -based state assessment resembles what officials in some states called “focused monitoring,” 
states were not consistent in their terminology. This report uses “performance-based state assessment” for 
processes where monitoring focused on low-performing sites, whether or not it focused on a few priority 
indicators. 



45 




Table 3. Definitions used to classify process types in 20 states 

Performance-based assessment 

Performance-based The state assumed primary responsibility for reviewing and interpreting data on 
state assessment compliance and/or outcomes and for identifying problems. This process 

involved a data collection visit by a state team or state contractor, examination 
of information captured by state databases, or both. 

Monitoring units were chosen based on specified performance criteria. In the 
selection process, some states first sorted monitoring units into peer groups and 
then ranked their performance. (Some states selected additional monitoring units 

randomly or permitted them to self-select.) 

Schedule-based assessment 

Schedule-based The state assumed primary responsibility for reviewing and interpreting data on 

state assessment compliance and/or outcomes and for identifying problems. This process 

involved a data collection visit by a state team or state contractor, examination 
of information captured by state databases, or both. 

All monitoring units participated, continuously, annually, or cyclically (every x 
years). If a process was designed for monitoring units of a certain size or 
program type, the process was classified as schedule-based state assessment so 
long as all monitoring units of that size or type were subject to it. 

The monitoring unit, or a regional intermediary, was required to collect, review, 
and interpret data on compliance and/or outcomes and identify the problems that 
required action. If the monitoring unit entered data in an electronic database that 
flagged areas of noncompliance or underperformance, it was included in this 
category. If the monitoring unit merely collected data for the state to review and 
interpret, the process was considered a schedule-based state assessment. 

All monitoring units participated, continuously, annually, or cyclically (every x 
years). If a process was designed for monitoring units of a certain size or 
program type, the process was classified as schedule-based local assessment so 
long as all monitoring units of that size or type were subject to it. 

Schedule-based The process involved a sequence of schedule-based local assessment followed 

hybrid assessment by a site visit and/or some other form of additional data collection by a state 

team or state contractor. In some cases, the site visit was labeled a “verification” 
or “validation” visit, signifying that its primary purpose was to check the local 
assessment findings; in other cases, the state visit had a broader agenda. 

All monitoring units participated, continuously, annually, or cyclically (every x 
years). If a process was designed for monitoring units of a certain size or 
program type, the process was classified as schedule-based hybrid assessment 

so long as all monitoring units of that size or type were subject to it. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 



Schedule-based 
local assessment 
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Overview of Part B Approaches to Monitoring 

The first section below looks more closely at the process types observed for Part B in 2004-05 
and 2006-07 and how often the processes were initiated. It is followed by an examination of 
whether and how states combined Part B monitoring processes of different types — perfonnance 
based and schedule based — to create a monitoring system in each time period. 

Part B Monitoring Processes 

This section begins with an examination of Part B monitoring process types and how often they 
were initiated. 

Part B Monitoring Process Types 

In 2004-05, the 20 Part B monitoring systems included 34 distinct monitoring processes. Table 4 
shows how those processes were distributed by process type. All four process types were 
observed. Ten (29%) of the Part B monitoring processes were performance-based state 
assessments, and 24 (71%) were schedule based. The latter category included nine schedule- 
based local assessment processes (26% of all processes), eight schedule -based hybrid processes 
(24%), and seven schedule-based state assessment processes (21%). 

Table 4. Number of state Part B monitoring processes that used various process types: 
2004-05 and 2006-07 



Process type 


2004-05 
(N= 34) 


2006-07 
(N= 32) 


Performance based 


Perfonnance-based state assessment 


10 


11 


Schedule based 


Schedule-based local assessment 


9 


7 


Schedule-based state assessment 


7 


7 


Schedule-based hybrid assessment 


8 


7 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

In 2004-05, a total of 6 of the 10 perfonnance-based state assessment processes used a short list 
of indicators or priority areas (four or fewer) to identify LEAs with performance issues. A 
process might focus exclusively on dropout rates, for example, or on a couple of indicators such 
as placement in least restrictive environment (LRE) and graduation rates. In the remaining four 
performance-based state assessment processes, a dozen indicators or more were used to identify 
LEAs. While all Part B monitoring systems using this type of process focused on 
underperforming LEAs, three Part B monitoring systems also included a few other LEAs, such 
as volunteers or LEAs chosen randomly. 

Twelve of the 16 schedule-based local and state assessment processes and all 8 of the schedule- 
based hybrid assessment processes examined many indicators, anywhere from a dozen to 
hundreds. Four of the eight schedule-based hybrid assessment processes were designed to take 
multiple years to fully implement. For example, one process began with a local assessment by 
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the LEA during Year 1, culminating in an action plan that addressed both compliance and 
performance issues. In Year 2, there was a state “validation” visit, followed by a period of plan 
implementation and progress reporting. In contrast, another process called for local assessment, 
validation, and correction of compliance all to occur within a single year. 

In 2006-07, the 20 Part B monitoring systems included 32 distinct processes. The types of 
processes observed were consistent with the observations for the previous time period. Eleven 
processes (34%) were performance-based state assessments, and 2 1 (66%) were schedule-based 
processes. There were seven instances of each type of schedule-based process — local 
assessment, state assessment, and hybrid assessment. 

Initiation of Part B Monitoring Processes 

Table 5 shows another aspect of Part B monitoring processes — how often each LEA was 
required to initiate the monitoring process. While monitoring processes could have been 
categorized by “how often LEAs were monitored” or “how long monitoring lasted,” using these 
categories was potentially problematic. For instance, how long and how often an LEA was 
monitored depended partly on how quickly and how satisfactorily it responded to the 
identification of its compliance or performance issues. In addition, duration of monitoring 
depended on how the Part B monitoring system structured the process — whether the monitoring 
process had multiple phases, for example — and whether it targeted outcome issues, which could 
take longer to address than noncompliance. Therefore, it was more precise to categorize Part B 
monitoring processes based on when monitoring was initiated. 

Table 5. Number of state Part B monitoring processes in which monitoring was 
initiated at various frequencies: 2004-05 and 2006-07 



Frequency of initiation 


2004-05 

(A=34) 


2006-07 
(N= 32) 


As perfonnance dictated 


10 


11 


Annual 


9 


10 


Cyclical 

Every 2-3 years 


3 


3 


Every 4 years or more 


12 


8 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Table 5 sorts frequency of initiation into three broad categories. Ten of the 34 Part B monitoring 
processes (29%) were initiated “as performance dictated” in 2004-05. This category contains the 
10 Part B monitoring processes that were performance-based state assessments. In these cases, a 
monitoring process was initiated only when an LEA was selected because of performance on 
specified indicators. 17 How long the monitoring process lasted depended primarily on how long 
it took the LEA to satisfy state officials that it no longer required special attention. Because 
many of the performance-based state assessments targeted outcome issues that could not be 
corrected quickly, such a monitoring process could last several years. 



17 Systems using this approach reviewed their indicators and made decisions each year regarding which LEAs 
would be selected. However, from the standpoint of an individual LEA, the process was not annual. 
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In 2004-05, nine other Part B monitoring processes (26%) were initiated annually. That is, LEAs 
were expected to start those monitoring processes anew every single year. For example, in one 
Part B monitoring process, an LEA completed a schedule-based local assessment every year, 
using a guide developed by the state, followed by an improvement plan addressing problem 
areas. 

Fifteen of the 34 Part B monitoring processes (44%) were initiated “cyclically,” a term adopted 
from common state usage. Cyclical monitoring means that the state agency monitored a subset of 
LEAs each year, so that by the end of the cycle, all LEAs had participated in the monitoring 
process at least once. For example, a 3-year cycle meant that each LEA began the monitoring 
process every 3 years. In 2004-05, a total of 12 of the 15 Part B monitoring processes with 
cyclical monitoring (80%) had a cycle of 4 years or more; the remaining three Part B monitoring 
processes (20%) had a cycle of 2 to 3 years. 

The duration of cyclical monitoring processes, like the other processes, depended partly on how 
quickly LEAs addressed their identified problems. In addition, several of the cyclical processes 
were specifically designed to have steps or phases that spanned 2 years or more (such as local 
assessment, verification, and implementation). In 2004-05, a total of 6 of the 15 cyclical Part B 
monitoring processes had this feature. 

In 2006-07, a total of 1 1 of the 32 Part B monitoring processes (34%) were initiated as 
performance dictated; 10 of the 32 Part B monitoring processes (31%) were annual; and 1 1 of the 
32 Part B monitoring processes (34%) were cyclical. Of the cyclical processes, three were 
designed to last 2 to 3 years, and eight were designed to last 4 years or more. 

Part B Monitoring Systems 



This section shifts attention from the individual processes described above to looking at Part B 
monitoring systems, which in some instances consisted of a single process and in others, multiple 
processes used in combination. Interviewees in multi-process systems mentioned various factors 
that led to multiple processes. First, as noted earlier, different types of processes were viewed as 
having different advantages. (For example, perfonnance-based state assessment can concentrate 
resources on high-need districts, although it does not reach every LEA. Both schedule-based 
local assessment and schedule-based hybrid assessment are believed to promote greater local 
investment, although they provide less control over procedures than schedule-based state 
assessment.) Thus, state agency staff sometimes reported that they viewed different types of 
monitoring processes as complementary, i.e., compensating for the potential disadvantages of a 
single-process type. Second, state monitoring systems sometimes used separate processes to 
address different types of indicators (e.g., compliance vs. outcomes) or different types of 
monitoring units (e.g., big districts vs. small districts). Third, state monitoring systems that were 
undergoing transition sometimes had older and newer processes operating side by side for some 
period of time. The project team could not categorize the reasons state agency staff chose single 
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1 8 

or multiple processes, however, because the information was not consistently available. In the 
sections below, the discussion is limited to what Part B monitoring systems actually did. 

Part B Monitoring Systems and Process Types 

In 2004-05 and 2006-07, there were 10 single-process Part B monitoring systems and 10 multi- 
process Part B monitoring systems. During both time periods, each single-process system used 
either perfonnance-based state assessment or one of the three types of schedule-based 
assessment (local, state, or hybrid). All four types of assessment were also found in the multi- 
process systems in both time periods. The multi-process Part B monitoring systems used either a 
combination of schedule-based processes or a combination of perfonnance- and schedule-based 
assessment processes. 19 No multi-process systems were observed to rely exclusively on 
performance-based assessments in either time period. Looking across all 20 Part B monitoring 
systems, in both periods, half or more used only schedule-based processes — 1 1 systems (55%) in 
2004-05 and 10 systems (50%) in 2006-07. 

In 1 1 of the 20 states, the number or the type of processes that made up their Part B monitoring 
systems changed between 2004-05 and 2006-07, or both kinds of change occurred. There was no 
identifiable pattern to these changes; for example, both increases and decreases in the number of 
processes were observed. ~ Nine Part B monitoring systems changed neither the number nor 
type of processes. 

Part B Monitoring Systems and Monitoring Initiation 

Another aspect of Part B single- and multi-process systems was how frequently monitoring was 
initiated at the system level. Individual monitoring processes have their own timetable (annual, 
cyclical, or dictated by performance), but, within a multi-process system, the timing of the 
individual processes may differ. For example, a Part B monitoring system might require LEAs to 
participate in both a cyclical process and an annual process. 

In 2004-05 and 2006-07, each single-process Part B monitoring system initiated monitoring on 
an annual, cyclical, or as performance-dictated basis. In 2004-05, 7 of the 10 single-process 
systems (70%) initiated that process cyclically. In 2006-07, a total of 3 of the 10 single-process 
Part B monitoring systems (30%) initiated their process cyclically." The 10 multi-process Part B 
monitoring systems also included processes that initiated monitoring annually, cyclically, and as 
performance dictated in both time periods, in various combinations. In 2004-05, a total of 6 of 
the 10 multi-process systems (60%) included at least one annual process, and 4 (40%) included 
at least one cyclical process (but no annual ones). In 2006-07, five of the multi-process systems 
(50%) had at least one annual process, and the other five had at least one cyclical process (but no 
annual ones). None of the multi-process systems initiated monitoring only as perfonnance 



18 The documentation that site visitors reviewed was primarily limited to 2004-05 and 2006-07 and did not always 
cover the history or rationale for the approaches in use. In addition, current staff and other informants sometimes 
had not been involved when decisions were made. 

19 See footnote 13 on page 41. 

20 See footnote 13 on page 41. 

21 See footnote 13 on page 41. 
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indicated in either time period. Looking across all 20 Part B monitoring systems, 1 1 (55%) in 
2004-05 used at least one cyclical process, but no annual processes; 8 Part B monitoring systems 
(40%) used at least one cyclical process, but no annual processes in 2006-07.“ 

Overview of Part C Approaches to Monitoring 

Project staff used the same approach that was used to examine Part B monitoring systems and 
processes to examine Part C monitoring systems and processes in 2004-05 and 2006-07. 

Part C Monitoring Processes 



This section begins with an examination of Part C monitoring process types and then looks at 
how frequently individual Part C monitoring processes were initiated. 

Part C Monitoring Process Types 

In 2004-05, the 20 Part C monitoring systems included 24 distinct monitoring processes. As seen 
in table 6, a total of 2 monitoring processes (8%) were performance-based state assessments, and 
22 (92%) were schedule-based processes. The 22 schedule-based processes included 18 state 
assessments (75% of all processes), 3 local assessment processes (13%), and 1 hybrid assessment 
process (4%). 

Table 6. Number of state Part C monitoring processes that used various process types: 
2004-05 and 2006-07 



Process type 


2004-05 
(N= 24) 


2006-07 

(A=28) 


Performance based 


Perfonnance-based state assessment 


2 


4 


Schedule based 


Schedule-based local assessment 


3 


5 


Schedule -based state assessment 


18 


16 


Schedule-based hybrid assessment 


1 


3 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

In 2004-05, some 15 of the 18 schedule-based state assessment processes (83%) for Part C 
featured a site visit by a state agency team that could include state personnel, contractors, and 
others, such as parents or peers from other EIS programs. Two of the other three schedule-based 
state assessment processes utilized MISs that provided up-to-date child-specific infonnation; in 
lieu of site visits, monitoring staff carried out the process primarily through frequent examination 
of reports from the databases. 

In 2006-07, the 20 Part C monitoring systems studied included 28 distinct monitoring processes. 
Four of these Part C monitoring processes (14%) were perfonnance-based state assessments, and 
the remaining 24 Part C monitoring processes (86%) were schedule based. Among the schedule- 



22 See footnote 13 on page 14. 
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based processes, 16 (57% of all types) were state assessments; 5 were local assessment processes 
(18%); and 3 were hybrid assessment processes (11%). 

Initiation of Part C Monitoring Processes 

Table 7 shows the frequency with which each Part C monitoring process was initiated in 2004-05 
and 2006-07. Note that, for Part C, one additional frequency category was included that did not 
exist for Part B because it was found that some Part C monitoring processes were initiated more 
frequently than annually. These Part C monitoring processes were classified as “continuous” 
monitoring, borrowing the terminology used by some of the states. In practice, continuous 
monitoring meant that state agency staff routinely reviewed various indicators anywhere from 
monthly to semiannually, but could follow the data for individual EIS programs more often if the 
circumstances dictated. 

Table 7. Number of state Part C monitoring processes in which monitoring was 
initiated at various frequencies: 2004-05 and 2006-07 



Frequency of initiation 


2004-05 

(JV=24) 


2006-07 
(N= 28) 


As performance dictated 


2 


4 


Continuous 


4 


5 


Annual 


6 


9 


Cyclical 

Every 1 8 months to 3 years 


9 


7 


Every 4 years or more 


3 


3 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

In 2004-05, a total of 2 of the 24 Part C monitoring processes (8%) were initiated as EIS 
performance dictated; these were the two performance-based state assessment processes (see 
table 6). Four of the 24 Part C monitoring processes (17%) were continuous — all of them 
schedule-based state assessments. For each of the continuous processes, the state agency had 
access to one or more MISs that allowed it to examine EIS program compliance and performance 
and flag problems more often than annually. Six of the 24 Part C monitoring processes (25%) 
were initiated annually. The remaining 12 Part C monitoring processes (50%) were initiated 
cyclically. Nine of the cycles were initiated every 18 months to 3 years, while three had a cycle 
of 4 years or more. 

In 2006-07, a total of 4 of the 28 Part C monitoring processes (14%) were initiated as dictated by 
EIS program performance (performance-dictated); 5 of the 24 Part C monitoring processes 
(18%) were continuous; and 9 of the 24 Part C monitoring processes (32%) fell into the annual 
category. The remaining 10 Part C monitoring processes (36%) were cyclical; 7 processes 
initiated a cycle every 18 months to 3 years, and 3 initiated a cycle every 4 years or more. As in 
2004-05, all of the continuous processes were schedule-based state assessments. 
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Part C Monitoring Systems 



This section turns from individual monitoring processes to an examination of Part C monitoring 
systems, which in some instances consisted of a single process and, in others, multiple processes 
used in combination. 

Part C Monitoring Systems and Process Types 

In 2004-05, there were 17 single-process and 3 multi-process Part C monitoring systems. In 
2006-07, there were 13 single-process and 7 multi-process Part C monitoring systems. In 2004- 
OS, the single-process systems used either a local or state schedule-based assessment; none used 
performance-based or hybrid schedule-based assessments. In 2006-07, all four types of 
assessment processes were used by the single-process systems. 23 

In both 2004-05 and 2006-07, the multi-process systems used all four types of assessment 
processes. These Part C monitoring systems either used schedule-based processes only or 
combined both perfonnance-based and schedule-based processes. 24 None of the multi-process 
systems in either time period included only performance-based assessments. 

In 7 of the 20 Part C monitoring systems, the number or the type of processes that made up the 
systems changed between 2004-05 and 2006-07, or both kinds of changes occurred; changes in 

25 

the number of monitoring processes went in both directions, including increases and decreases. 
Thirteen Part C monitoring systems changed neither the number nor type of monitoring 
processes. 

Part C Monitoring Systems and Monitoring Initiation 

As noted earlier, individual monitoring processes have their own timetable (annual, cyclical, or 
dictated by performance), but within a multi-process system, the timing of the individual 
processes may differ. In 2004-05, the 17 Part C single-process systems initiated monitoring 
continuously, annually, or cyclically; none of these Part C monitoring systems initiated 
monitoring on the basis of perfonnance. Twelve of the 17 single-process systems (71%) initiated 
that process cyclically. In 2006-07, Part C single-process systems initiated monitoring on a 

continuous, annual, cyclical, or perfonnance-dictated basis. During this time period, 6 of the 13 

26 

single-process systems (46%) initiated that process cyclically." 

In 2004-05, the three Part C multi -process systems included processes that initiated monitoring 
continuously, annually, or as performance dictated, in various combinations. They did not use 
cyclical monitoring processes. Each of these Part C monitoring systems had at least one 
continuous or annual process, so that none of the three systems initiated monitoring only as 
performance dictated. In 2006-07, the seven Part C multi-process systems included processes 
that initiated monitoring continuously, annually, cyclically, and as performance dictated. As in 



23 See footnote 13 on page 41. 

24 See footnote 13 on page 41. 

25 See footnote 13 on page 41. 

26 See footnote 13 on page 41. 
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the earlier period, none initiated monitoring only as performance dictated. All seven multi- 
process systems in 2006-07 had at least one continuous, annual, or cyclical process; four of the 
seven multi-process systems (57%) had at least one annual process, but no continuous 
processes. 27 

Looking across all 20 Part C monitoring systems, 12 (60%) in 2004-05 used at least one cyclical 
process, but no annual or continuous processes; 7 Part C monitoring systems (35%) used at least 
one cyclical process, but no annual or continuous processes in 2006-07. 28 

Summary for Overview of Approaches to Monitoring 

The preceding section has provided an overview of Part B and Part C monitoring approaches in 
the 20 states — focusing on a few broad attributes that were observable across processes and 
systems, regardless of how they implemented the components in the framework. Overall, in the 
20 states in the study, 58 monitoring processes were identified (34 processes for Part B, 24 
processes for Part C) in 2004-05 and 60 monitoring processes (32 processes for Part B, 28 
processes Part C) in 2006-07. 

Monitoring processes could be classified into four types, which were used singly or in 
combination by Part B and Part C monitoring systems to carry out states’ monitoring 
responsibilities. With performance-based state assessment, only LEAs or EIS programs that met 
certain perfonnance criteria were monitored. In these processes, monitoring was initiated as 
performance dictated. The three other types of processes — schedule-based local assessment, 
schedule-based state assessment, and schedule -based hybrid assessment — were universally 
applied. The three schedule-based process types differed in terms of what entity (the LEA, the 
state, or both) took primary responsibility for identifying problems. Schedule-based processes 
also varied as to whether they were initiated cyclically (i.e., every so many years), annually, or 
continuously. In the 20 states, continuous monitoring — that is, monitoring that was initiated 
more than once a year — was observed only in the state monitoring systems for Part C. 

In 2004-05 and 2006-07, there was an even split — 10 each — in single and multi-process Part B 
monitoring systems. In both time periods, both single-process and multi -process Part B 
monitoring systems included all four types of assessments. Multi-process systems used various 
combinations of processes — either all schedule-based or a mix of schedule-based and 
performance-based assessment. Almost one-half of the Part B monitoring systems (9 of the 20, 
or 45%) changed their number of monitoring processes between the two site visits. 

In both time periods, Part B single-process systems included processes that initiated monitoring 
on an annual, cyclical, or as performance dictated basis. The Part B multi-process systems also 
included processes that initiated monitoring annually, cyclically, and as performance dictated in 
both time periods; all had at least one process that was initiated on an annual or cyclical basis. 



27 See footnote 13 on page 41. 

28 See footnote 13 on page 41. 
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In 2004-05 and 2006-07, over half of Part C monitoring systems (17 of 20 and 13 of 20, 
respectively) were single-process. In 2004-05, none of the single-process systems used 
performance-based or schedule-based hybrid assessments; in 2006-07, all four types of 
assessment processes were observed. In 2004-05, there were three Part C multi-process systems, 
and in 2006-07, there were seven multi-process systems. During both time periods, none of the 
multi-process systems used only perfonnance-based assessments; they either used schedule- 
based processes alone or combined them with perfonnance-based processes. In the period 
between the two site visits, 7 of the 20 Part C monitoring systems (35%) changed either their 
number or type of processes. 

In 2004-05 and 2006-07, Part C single-process systems initiated monitoring on a continuous, 
annual, or cyclical basis. Only in 2006-07 did any single-process system report initiating 
monitoring on a performance basis. States with multi-process systems included continuous, 
annual, and performance-dictated monitoring processes in both time periods; processes that 
initiated monitoring cyclically were observed in 2006-07. None of the multi-process systems in 
either time period initiated monitoring only on a performance-indicated basis. 



Mapping Onto the Framework Components 

The previous section discussed states’ overall approaches to monitoring, such as whether they 
had Part B and Part C monitoring systems that were made up of a single process or multiple 
processes and whether they selected LEAs/EIS programs for monitoring using a perfonnance- 
based approach or a schedule-based approach. This section continues to discuss states’ 
approaches to monitoring, focusing on the nature and design of states’ monitoring processes and 
how those processes mapped onto the framework components. 

Findings are presented separately for Part B and Part C. Both the Part B and Part C sections 
begin by describing how many of the Part B or Part C monitoring systems and processes 
included each of the five monitoring components (i.e., Problem Identification, Problem 
Investigation, Corrective Action and Enforcement, Improvement Planning and Implementation, 
and Reassessment) and how the components were combined. Both the Part B and Part C sections 
then discuss each component in turn and describe the various strategies that states used in their 
monitoring processes to carry out that component in 2004-05 and 2006-07. All of the data 
discussed in this section come from the site visit profiles that were developed after each round of 
site visit data collection. As discussed in chapter 4, the state profiles were developed as a means 
of condensing the vast amount of descriptive information that was collected during the site visit 
interviews and documentation reviews. 

Part B Monitoring Processes 

This section presents findings about states’ Part B monitoring processes for each of the five 
framework components. 
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Table 8 presents the number of Part B monitoring systems and processes in 2004-05 and 2006-07 
that included each of the components. 29 

Table 8. Number of state Part B monitoring systems and processes that included the 
five framework components: 2004-05 and 2006-07 





2004-05 


2006-07 


Monitoring component 


Systems 

(jV=20) 


Processes 

((V=34) 


Systems 

(A=20) 


Processes 
(N= 32) 


Problem Identification 


20 


34 


20 


32 


Problem Investigation 


10 


16 


13 


17 


Corrective Action and Enforcement 


20 


32 


20 


32 


Improvement Planning and Implementation 


16 


23 


16 


21 


Reassessment 


6 


11 


5 


9 



NOTE: Numbers do not sum to totals because state monitoring systems and processes could include more than one 
component. The data referring to systems present findings for the state monitoring systems overall. States’ 
monitoring systems could be composed of a single set of activities (i.e., a single monitoring process) or multiple sets 
of activities (i.e., multiple monitoring processes). The data referring to processes present findings for the monitoring 
processes that made up the state monitoring systems. If the Part B monitoring system had more than one monitoring 
process, a component was considered present if at least one process included that component. If the Part B 
monitoring system only had one monitoring process, then the component had to be part of that process for the 
component to be considered present in the state’s monitoring system. States’ monitoring systems and processes may 
have included additional activities that were not captured by the site visit interviews and ratings. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Table 9 displays how states combined the framework components in their Part B monitoring 
processes. In both 2004-05 and 2006-07, the Part B monitoring processes had between two and 
five components, with the Problem Identification component being present in all processes. Four 
of the 34 Part B monitoring processes (12%) in 2004-05 and 4 of the 32 Part B monitoring 
processes (13%) in 2006-07 included all five of the framework components. Another 11 Part B 
monitoring processes (32%) in 2004-05 and 14 Part B monitoring processes (44%) in 2006-07 
included four components, with the most common combination of components in both years 
being Problem Identification, Problem Investigation, Corrective Action and Enforcement, and 
Improvement Planning and Implementation (i.e., all components except Reassessment). Fourteen 
Part B monitoring processes (41%) in 2004-05 and seven Part B monitoring processes (22%) in 
2006-07 had three components, with the most common combination of components being 
Problem Identification, Corrective Action and Enforcement, and Improvement Planning and 
Implementation. The remaining five Part B monitoring processes (15%) in 2004-05 and seven 
Part B monitoring processes (22%) in 2006-07 included two components, Problem Identification 
and Corrective Action and Enforcement. There were no Part B monitoring processes with a 
single component in either time period. 



29 If the state monitoring system had more than one process, a component was considered present in the state’s 
monitoring system if at least one process included that component. If the state monitoring system only had one 
monitoring process, then the component had to be part of that process for it to be considered present in the state’s 
monitoring system. 
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Table 9. 



Number of state Part B monitoring processes that included various numbers of 
the five framework components: 2004-05 and 2006-07 



Components 


2004-05 

(#=34) 


2006-07 

(#=32) 


Five components 


4 


4 


Four components 


11 


14 


Problem Identification, Problem Investigation, Corrective Action and 
Enforcement, Improvement Planning and Implementation 


6 


11 


Problem Identification, Problem Investigation, Improvement Planning and 
Implementation, Reassessment 


2 


0 


Problem Identification, Problem Investigation, Corrective Action and 
Enforcement, Reassessment 


1 


2 


Problem Identification, Corrective Action and Enforcement, Improvement 
Planning and Implementation, Reassessment 


2 


1 


Three components 


14 


7 


Problem Identification, Problem Investigation, Corrective Action and 
Enforcement 


3 


0 


Problem Identification, Problem Investigation, Improvement Planning and 
Implementation 


0 


0 


Problem Identification, Corrective Action and Enforcement, Improvement 
Planning and Implementation 


9 


5 


Problem Identification, Corrective Action and Enforcement, Reassessment 


2 


2 


Two components 


5 


7 


Problem Identification, Corrective Action and Enforcement 


5 


7 


Problem Identification, Improvement Planning and Implementation 


0 


0 


One component 


0 


0 


Problem Identification 


t 


t 



f Not applicable. 



NOTE: Only those combinations of components that were observed to occur in states’ monitoring processes in 
either 2004-05 or 2006-07 are presented in this table. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

The remainder of this section presents findings for each of the framework components for the 
Part B monitoring processes. First, a brief summary of the component and the number of the Part 
B monitoring systems and processes that included the component in each year (i.e., 2004-05 and 
2006-07) is presented. The rest of the discussion is then devoted to describing the various ways 
in which states’ approached that component with respect to their Part B monitoring processes. 

Problem Identification: Part B Monitoring Processes 

According to the framework presented in chapter 3 of this report, Problem Identification involves 
determining whether LEAs/EIS programs are in compliance with IDEA regulations and/or are 
performing at satisfactory levels on outcome indicators. Problem Identification includes: (1) 
Indicator and Target Setting, which involves identifying and defining indicators and establishing 
targets for those indicators; (2) Indicator Data Collection and Analysis, which involves collecting, 
preparing, and analyzing data for assessing perfonnance on identified indicators; and (3) Problem 
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Detection, which involves comparing performance on a specified indicator to the target established 
for that indicator. 

As shown in table 8 and noted above, in both 2004-05 and 2006-07, all 20 of the Part B monitoring 
systems (100%) and all of the Part B monitoring processes (34 of the 34 Part B monitoring processes 
in 2004-05 and 32 of the 32 Part B monitoring processes in 2006-07) included a Problem 
Identification component. 

Discussed below is how states approached Problem Identification in their Part B monitoring 
processes, including how they selected indicators and targets, the data sources they used to identify 
problems, who collected the data, and how the problems were actually identified. 

Selecting Indicators and Targets 

This section discusses how indicators were selected to identify problems, the types of indicators 
that were used (i.e., compliance or outcome indicators), and whether a large set or a small set of 
indicators was chosen. 

Selection of indicators . In 2004-05, in 21 of the 34 Part B monitoring processes (62%), state 
agency staff selected indicators to assess local performance with assistance from a state-level 
stakeholder committee (see table 10). For example, in one Part B monitoring process, during the 
stakeholder meetings, the state agency staff presented data to the stakeholder committee and 
reviewed performance in specific areas. The stakeholders then assisted the state agency staff in 
selecting priority areas for indicators and drafting indicators. In another Part B monitoring 
process, the state agency staff selected the indicators and then asked the stakeholders to provide 
input regarding how the indicators should be defined and measured. In 13 of the 34 Part B 
monitoring processes (38%), the state agency staff selected indicators to assess local 
performance without input from a state-level stakeholder committee and instead chose their 
indicators based on reviews of state and federal implementation guidelines. 

Table 10. Number of state Part B monitoring processes that used various approaches to 
select indicators: 2004-05 and 2006-07 



How indicators selected 


2004-05 
{N= 34) 


2006-07 
(N= 32) 


By state, with stakeholder input 


21 


8 


By state, without stakeholder input 


13 


8 


By state, based on State Perfonnance Plan/ Annual 
Perfonnance Plan (SPP/APR) 


t 


16 



| Not applicable. Because the 2004 amendments to IDEA, which require states to submit SPPs/APRs, were not 
enacted until December 2004, this approach was not used by states when selecting their indicators for 2004-05. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

In 2006-07, in 16 of the 32 Part B monitoring processes (50%), state agency staff selected 
indicators primarily based on the OSEP SPP/APR indicators (see table 10). In 14 of the 16 Part B 
monitoring processes (88%) that used the SPP/APR indicators, state agency staff consulted with 
a state-level stakeholder committee in making this decision. In addition, in 9 of these Part B 
monitoring processes (56%), state agency staff reported that, when deciding to use the SPP/APR 
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indicators, they also took into consideration state regulations and may have included additional 
indicators in order to meet specific state-level requirements. In the remaining 16 Part B 
monitoring processes (50%), indicators were selected by the state agency staff. In half of these 
Part B monitoring processes (8 processes or 50%), state agency staff reported that they sought 
input from a state-level stakeholder committee; in the remaining Part B monitoring processes (8 
processes or 50%), this was not the case. 

Selection of compliance indicators and targets . Overall, 29 of the 34 Part B monitoring processes 
(85%) in 2004-05 and 28 of the 32 Part B monitoring processes in 2006-07 (88%) included 
compliance indicators to assess whether LEAs were in compliance with the federal regulations 
set forth in IDEA Part B (see table 11). Nine of the 34 Part B monitoring processes (26%) in 
2004-05 and 8 of the 32 Part B monitoring processes (25%) in 2006-07 used only compliance 
indicators. 

Table 11. Number of state Part B monitoring processes that used compliance and 
outcome indicators: 2004-05 and 2006-07 



Indicator types and combinations 


2004-05 

(tV=34) 


2006-07 
(N= 32) 


Indicator types 


Compliance 


29 


28 


Outcome 


25 


24 


Indicator combinations 


Compliance only 


9 


8 


Outcome only 


5 


4 


Both compliance and outcome 


20 


20 



NOTE: Numbers do not sum to totals for indicator types because state monitoring could use more than one type of 
indicator. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

30 

The targets for compliance indicators were generally 0 or 100 percent, as required by law. 
However, 6 of the 29 Part B monitoring processes (21%) in 2004-05 and 5 of the 28 Part B 
monitoring processes (18%) in 2006-07 that included compliance indicators either had targets 
below 100 percent, or the detennination of noncompliance was considered to be subjective (i.e., 
the point at which identified issues officially became problems was left up to the discretion of the 
staff member conducting the monitoring activities). These Part B monitoring processes focused 
more on systemic noncompliance — noncompliance that was recurrent or part of a pattern, rather 
than an isolated failing. For example, in one Part B monitoring process, targets were set at less 
than 100 percent, and LEAs were identified as out of compliance when a certain percentage of 
the student files reviewed were deficient on the same indicator area (e.g., 20% or more of the 
student files reviewed showed no evidence that student evaluations occurred within 60 days of 
parental consent for an evaluation). In another Part B monitoring process, individual instances of 



30 Whether the target was set at 0 or 100 percent depended upon how the indicator was worded. For example, 
“Percentage of IFSPs completed within 45-day timeline” would have a target of 100 percent, whereas 
“Percentage of IEPs that lacked a parent signature” would have a target of 0 percent. 
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noncompliance were noted (e.g., an IEP was missing a signature), but it was up to the state 
agency staff to decide if the individual instances reached the level of systemic noncompliance. 

Selection of outcome indicators and targets . As shown in table 11, 25 of the 34 Part B 
monitoring processes (74%) in 2004-05 and 24 of the 32 Part B monitoring processes (75%) in 
2006-07 included outcome indicators (e.g., indicators related to student perfonnance such as 
graduation rates or scores on reading or math assessments). Five of the 34 Part B monitoring 
processes (15%) and 4 of the 34 Part B monitoring processes (13%) included only outcome 
indicators. 

Sixteen of the 25 Part B monitoring processes (64%) in 2004-05 and 5 of the 24 Part B 
monitoring processes (21%) in 2006-07 that included outcome indicators did not have targets for 
these indicators. 

Number of indicators included . Overall, 24 of the 34 Part B monitoring processes (71%) in 2004- 
05 and 22 of the 32 Part B monitoring processes (69%) in 2006-07 included a large set of 
compliance and/or outcome indicators (i.e., anywhere from a dozen or so to hundreds in some 
cases) on which to focus and assess local performance (see table 12). The remaining 10 Part B 
monitoring processes (29%) in 2004-05 and 10 Part B monitoring processes (31%) in 2006-07 
focused on a small set of compliance and/or outcome indicators (i.e., between one and four 
indicators or priority areas). These indicators were chosen to reflect particular concerns (e.g., 
LRE, disproportionality, child find), often in consultation with a state-level stakeholder 
committee. 

Table 12, Number of state Part B monitoring processes that used either a large or small 
set of compliance and/or outcome indicators: 2004-05 and 2006-07 





2004-05 


2006-07 


Number of indicators used 


(/V=34) 


(N= 32) 


Small set of indicators 


10 


10 


Large set of indicators 


24 


22 



NOTE: A small set of indicators was defined as between one and four indicators or priority areas; a large set of 
indicators was defined as five or more indicators or priority areas. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Problem Identification Data Sources and Collection 

This section describes the data sources used to assess local performance on the indicators and 
who typically was involved in the data collections — either the state agency staff, the LEA staff, 
or both. 

Problem Identification data sources . States’ Part B monitoring processes used data from different 
sources to assess performance on indicators (see table 13). The most common source of data was 
existing state or local data, such as data profiles or data for section 618 of IDEA (e.g., 



60 




•5 1 

suspension/expulsion data, racial/ethnic data to assess disproportionality) to assess local 
performance; this data source was used by 25 of the 34 Part B monitoring processes (74%) in 
2004-05 and 26 of the 32 Part B monitoring processes (81%) in 2006-07. Another common 
source of data was record reviews, used by 23 of the 34 Part B monitoring processes (68%) in 
2004-05 and 23 of the 32 Part B monitoring processes (72%) in 2006-07. Record reviews were 
done to assess local performance on compliance indicators. Interviews or forums/focus groups to 
obtain input from various stakeholders were used by 18 of the 34 Part B monitoring processes 
(53%) in 2004-05 and 16 of the 32 Part B monitoring processes (50%) in 2006-07. Other data 
sources included surveys (often parent surveys) and observation of students in their learning 
environments. 

Table 13. Number of state Part B monitoring processes that used various data sources to 
identify problems: 2004-05 and 2006-07 



Data source 


2004-05 

(jV=34) 


2006-07 
(N= 32) 


Record reviews 


23 


23 


Interviews/forums/focus groups 


18 


16 


Existing state/local data 


25 


26 


Surveys 


12 


12 


Student observations 


3 


2 



NOTE: Numbers do not sum to totals because state monitoring processes could use more than one data source. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

In 22 of the 34 Part B monitoring processes (65%) in 2004-05 and 23 of the 32 Part B 
monitoring processes (72%) in 2006-07, two or more of these sources of data were combined in 
order to identify problems in LEAs. For example, in one Part B monitoring process, state agency 
staff conducted record reviews, reviewed data profiles, conducted interviews with parents and 
LEA staff, and observed students in classrooms. All of the infonnation collected was then 
reviewed in order to assess the LEA’s performance on the indicators. In another Part B 
monitoring process, state agency staff reviewed LEA profiles and other local data, conducted 
parent surveys, interviewed teachers and other local administrators, and conducted record 
reviews. The remaining 12 Part B monitoring processes (35%) in 2004-05 and 9 Part B 
monitoring processes (28%) in 2006-07 used only one of these sources of data to identify 
problems in LEAs. 

Who collected data for Problem Identification ? As shown in table 14, LEA staff often 
participated in some manner in the collection of the data to identify problems, either alone or in 
conjunction with state agency staff. LEA staff were involved in data collection in 27 of the 34 
Part B monitoring processes (79%) in 2004-05 and 26 of the 32 Part B monitoring processes 



31 Section 618 of IDEA requires that data be reported by states each year to the Secretary of the U.S. Department of 
Education. For Part B, these data include: child count, educational environments, participation and performance 
on state assessments, dispute resolution, personnel employed, exiting (e.g., graduation, dropping out, etc.), and 
discipline. For Part C, these data include: child count, program settings, and reasons for exiting (e.g., completion 
of IFSP, eligibility for Part B services, withdrawal by a parent or guardian, etc.). In addition, the Secretary has the 
authority to collect “any other information that may be required by the Secretary.” 
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(81%) in 2006-07. In 8 of the 27 Part B monitoring processes (30%) in 2004-05 and 7 of the 26 
Part B monitoring processes (27%) in 2006-07, LEAs were required to convene a team or 
committee to assist with the data collection. These teams generally included parents, providers, 
and other local staff. For example, in one Part B monitoring process, LEAs were required to 
complete a self-assessment, which included reviews of 618 data and record reviews. Each LEA 
was required to assemble a stakeholder team to assist it in the data collection; the teams included 
parents, teachers, and local administrators. If the LEA staff were not involved in data collection, 
then data collection was completed solely by the state agency staff; this included 7 of the 34 Part 
B monitoring processes (21%) in 2004-05 and 6 of the 32 Part B monitoring processes (19%) in 
2006-07. 

Table 14. Number of state Part B monitoring processes in which data for identifying 
problems were collected by LEA staff and/or state agency staff: 2004-05 and 
2006-07 



Who collected the data? 


2004-05 

(A=34) 


2006-07 
(N= 32) 


LEA staff only 


10 


9 


State agency staff only 


7 


6 


Both LEA staff and state agency staff 


17 


17 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Identifying Problems 

This section describes how problems were identified for compliance and outcome indicators and 
who typically identified the problems, either the state agency staff, the LEA, or both. 

Identifying problems for compliance indicators . For 23 of the 29 Part B monitoring processes 
(79%) in 2004-05 and 23 of the 28 Part B monitoring processes (82%) in 2006-07 that included 
compliance indicators, identifying problems with regard to compliance-based indicators involved 
comparing the LEA’s performance on the compliance indicator to the target of 100 percent. If 
the LEA’s performance was less than the 100 percent target, then it was identified as a problem. 
As discussed previously, however, 6 of the 29 Part B monitoring processes (21%) in 2004-05 
and 5 of the 28 Part B monitoring processes (18%) in 2006-07 that included compliance 
indicators set targets that were less than 100 percent and identified compliance issues only when 
a certain percentage of the student records reviewed were found to be in noncompliance on the 
same indicator. In these Part B monitoring processes, LEAs were required to correct all areas of 
noncompliance found in the student records reviewed, but a formal finding of noncompliance 
requiring a CAP was issued only for those areas identified as systemic problems (e.g., recurring 
in 20% or more of the files reviewed). 

Identifying problems for outcome indicators . For those Part B monitoring processes that included 
outcome indicators and had targets for those indicators (9 of the 25 Part B monitoring processes 
or 36% in 2004-05 and 19 of the 24 Part B monitoring processes or 79% in 2006-07), problems 
were identified when the LEA did not meet the target. However, as noted earlier, 16 of the 25 
Part B monitoring processes (64%) in 2004-05 and 5 of the 24 Part B monitoring processes 
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(21%) in 2006-07 did not set targets for their outcome indicators. In these Part B monitoring 
processes, problems were identified (1) more subjectively by the state and/or LEA personnel 
who conducted the monitoring (e.g., they analyzed data and found LEA performance was below 
that of other LEAs in the state and not improving) or (2) by ranking LEA performance on the 
indicator(s) and using state-detennined criteria to select the lowest perfonning LEA(s). 

Who identified the problems ? Although LEA staff participated in the collection of data, in many 
Part B monitoring processes, it was the state agency staff who reviewed the data and identified 
which LEAs had compliance or outcome problems (see table 15). In 26 of the 34 Part B 
monitoring processes (76%) in 2004-05 and 25 of the 32 Part B monitoring processes (78%) in 
2006-07, the state agency staff were involved in some manner in identifying problems. More 
specifically, problems were identified strictly by state agency staff in 18 of the 34 Part B 
monitoring processes (53%) in 2004-05 and in 19 of the 32 Part B monitoring processes (59%) in 
2006-07. Both the state agency staff and the LEA staff identified problems in 8 of the 34 Part B 
monitoring processes (24%) in 2004-05 and 6 of the 32 Part B monitoring processes (19%) in 
2006-07. In the remaining Part B monitoring processes, problems were identified strictly by LEA 
staff, with oversight from the state agency (8 of the 34 Part B monitoring processes or 24% in 
2004-05 and 7 of the 32 Part B monitoring processes or 22% in 2006-07). 

Table 15, Number of state Part B monitoring processes in which problems were 
identified by state agency staff and/or LEA staff: 2004-05 and 2006-07 



Who identified problems? 


2004-05 

(A=34) 


2006-07 

(N=32) 


LEA staff only 


8 


7 


State agency staff only 


18 


19 


Both LEA staff and state agency staff 


8 


6 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Problem Investigation: Part B Monitoring Processes 



The previous section discussed approaches to identifying problems. According to the framework 
presented in chapter 3 of this report, once a problem is identified, Problem Investigation is 
needed to gather additional information or data to explain why the problem exists and what 
needs to be done to address it. 

As shown in table 8, a total of 10 Part B monitoring systems (50%) in 2004-05 and 13 Part B 
monitoring systems (65%) in 2006-07 included procedures for investigating problems that were 
identified in LEAs. Overall, 16 of the 34 (47%) Part B monitoring processes in 2004-05 and 17 
of the 32 (53%) Part B monitoring processes in 2006-07 included a Problem Investigation 
component. 

Discussed below is how states approached Problem Investigation in their Part B monitoring 
processes, including whether problem investigations were conducted by state agency staff or 
LEA staff, the nature of problem investigations, and the kinds of data and information used to 
investigate problems. 



63 




Who Conducted Problem Investigations? 

As shown in table 16, problem investigations were often conducted by state agency staff. In 11 
of the 16 Part B monitoring processes (69%) in 2004-05 and 9 of the 17 Part B monitoring 
processes (52%) in 2006-07, problem investigations were conducted strictly by state monitoring 
staff, sometimes with the assistance of a monitoring team convened specifically for this purpose. 
The teams often included other state agency staff, parents, and administrators from LEAs other 
than the one being monitored. In 5 of the 16 Part B monitoring processes (31%) in 2004-05 and 8 
of the 17 Part B monitoring process (47%) in 2006-07 that included a Problem Investigation 
component, LEA staff participated in problem investigations, either alone or in conjunction with 
state agency staff. When LEAs conducted problem investigations, they generally received some 
type of training, technical assistance, or guidance from the state, often in the form of worksheets, 
workbooks, and/or questions with standard probes. 

Table 16. Number of state Part B monitoring processes in which problems were 
investigated by state agency staff and/or LEA staff: 2004-05 and 2006-07 



Who investigated problems 


2004-05 
(N= 16) 


2006-07 
(A/=l 7) 


LEA staff only 


3 


6 


State agency staff only 


11 


9 


Both LEA staff and state agency staff 


2 


2 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Tailoring of Problem Investigations 

In 10 of the 16 Part B monitoring processes (63%) in 2004-05 and 10 of the 17 Part B 
monitoring processes (59%) in 2006-07, the same approach was used to investigate problems 
across LEAs, regardless of the nature of the problems identified. In the remaining six Part B 
monitoring processes (38%) in 2004-05 and seven Part B monitoring processes (41%) in 2006- 
07, the exact approach or approaches were based on the nature of the problem under 
investigation. For example, in one Part B monitoring process, the state monitoring teams tailored 
their approach to explore the hypotheses that a local team generated about why a particular 
problem existed. The approach could then be modified at any time, depending upon what 
additional information was uncovered during the investigation. In another Part B monitoring 
process, once the lowest performing LEAs were selected, the state monitoring teams that 
conducted the problem investigations examined the available data and then developed 
hypotheses about the LEA’s poor performance. These hypotheses then guided the nature of the 
problem investigations. 

Data and Information Used to Investigate Problems 

This section describes the data and information that were used to investigate problems in LEAs. 

Previously collected data or information . In 14 of the 16 Part B monitoring processes (88%) in 
2004-05 and 14 of the 17 Part B monitoring processes (82%), problem investigations included 
the review of data and/or information that had been previously collected for other purposes (see 
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table 17). The types of data and/or information reviewed included data collected for section 618 
of IDEA (including student assessment data), previous monitoring reports, and LEA policies and 
practices. For example, in one Part B monitoring process, after problems were identified during 
local on-site visits, dispute resolution data were examined to try to gain a better understanding of 
those problems. In another Part B monitoring process, state monitoring staff reviewed LEA data 
reports tailored to the problems under investigation. In other Part B monitoring processes, the 
data that were collected to identify problems were examined more closely or in different ways in 
order to get a better sense as to why the problem existed. 

Table 17, Number of state Part B monitoring processes that used previously collected 
and newly collected data to investigate problems: 2004-05 and 2006-07 



Data source 


2004-05 
(AM 6) 


2006-07 
(AM 7) 


Previously collected data or information 


14 


14 


Newly collected data 


10 


11 


Record review data 


10 


10 


Interviews/forums/focus groups 


10 


10 


Student observations 


3 


6 



NOTE: Numbers to do not sum to totals because state monitoring processes could use more than one data source. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Newly collected data . In 10 of the 16 Part B monitoring processes (63%) in 2004-05 and 11 of 
the 17 Part B monitoring processes (65%) in 2006-07, the investigations of problems included 
the collection and analysis of new data — mostly qualitative data — from multiple sources (see 
table 17). The types of new data collections commonly used by states for problem investigations 
included data collected from reviews of student records (records were selected either 
purposefully, based on the problems being investigated, or randomly); interviews/forums/focus 
groups with teachers, administrators, other service providers, parents, and students; and 
classroom or school observations. 

Addressing Identified Problems: Part B Monitoring Processes 

According to the framework presented in chapter 3 of this report, once a state identifies problems 
and conducts an investigation to explore why those problems exist, the next step is to address 
those problems. Two components from the framework are related to addressing identified 
problems: (1) Corrective Action and Enforcement, which is used when identified problems are 
related to noncompliance with procedural requirements or requirements to provide appropriate 
services in a timely manner, 32 and (2) Improvement Planning and Implementation, which is 
typically used when identified problems are more systemic (either locally or statewide), such as 
when there is pervasive noncompliance in certain areas, or the problems involve 
underperformance on selected outcome targets or benchmarks. 



32 As noted in the framework, “corrective action” has a specific meaning in IDEA. Findings of noncompliance must 
be corrected as soon as possible and in no case later than 1 year after identification. 
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In both 2004-05 and 2006-07, all 20 Part B monitoring systems (100%) had procedures in place 
to address the problem(s) identified through monitoring activities. In both 2004-05 and 2006-07, 
a total of 16 of the 20 Part B monitoring systems (80%) addressed identified problems through 
both Corrective Action and Enforcement and Improvement Planning and Implementation. The 
remaining four Part B monitoring systems (20%) addressed identified problems solely through 
Corrective Action and Enforcement. In 2004-05, a total of 32 of the 34 Part B monitoring 
processes (94%) included Corrective Action and Enforcement, and 23 of the 34 Part B 
monitoring processes (68%) included Improvement Planning and Implementation. In 2006-07, 
all 32 of the Part B monitoring processes (100%) included Corrective Action and Enforcement, 
and 21 of the 32 Part B monitoring processes (66%) included Improvement Planning and 
Implementation (see table 8). 

How the problems identified in LEAs were addressed by states’ Part B monitoring processes is 
presented below. In general, states included two activities: (1) the development of a plan 
outlining the identified problems that were serious enough to require some action by the LEA 
and (2) some type of follow-up to determine that the plan was implemented. 

Development of a Plan 

Generally, a written plan was required in response to findings of noncompliance or other 
identified problems. This section describes the development and nature of CAPs and 
improvement plans. 

Development of corrective action plans . CAPs were developed to address problems related to 
noncompliance. In the majority of Part B monitoring processes that included a Corrective Action 
and Enforcement component, CAPs were developed to address both individual student-level and 
systemic noncompliance (26 of the 32 Part B monitoring processes or 81% in 2004-05 and 28 of 
the 32 Part B monitoring processes or 88% in 2006-07). However, in 6 of the 32 Part B 
monitoring processes (19%) in 2004-05 and 4 of the 32 Part B monitoring processes (13%) in 
2006-07, CAPs were developed strictly to address instances of noncompliance with procedures 
and timelines at the individual student level. In these cases, if an individual student record was 
found to be noncompliant with either state or federal regulations, the LEA was required to 
address the noncompliance within a short period of time, typically 30 days or 45 days; systemic 
noncompliance was then addressed through improvement plans. For example, in one Part B 
monitoring process that used this approach, CAPs were developed for any identified problems 
that required immediate correction, that is, correction within 30 days of receipt of the on-site 
monitoring report. Typically, these were noncompliance issues related to individual child 
records. The LEA then had 30 days to implement the corrective action and submit proof of 
correction to the state agency. For problems that were more systemic in nature, defined as 
potentially affecting the quality of services and outcomes for all students with disabilities, LEAs 
were given 6 months to address the identified problems via an improvement plan. 

LEA staff typically developed CAPs (see table 18). In 25 of the 32 Part B monitoring processes 
(78%) in 2004-05 and 23 of the 32 Part B monitoring processes (72%) in 2006-07, the LEA staff 
were primarily responsible for developing CAPs. In 2 of the 32 Part B monitoring processes 
(6%) in 2004-05 and 4 of the 32 Part B monitoring processes (13%) in 2006-07, LEA staff and 
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state agency staff jointly developed CAPs. In the remaining 4 Part B monitoring processes (13%) 
in both 2004-05 and 2006-07, the state agency staff developed the CAPs. 33 

When LEA staff participated in the development of CAPs (27 of the 32 Part B monitoring 
processes in both 2004-05 and 2006-07), state agency staff often provided guidance, either by 
providing a specific template for LEAs to use, guidelines for the LEAs to follow (e.g., the CAP 
needed to include steps to be taken or a section to describe evidence of correction), or technical 
assistance. State agency staff provided guidance to LEAs in 13 of the 27 Part B monitoring 
processes (48%) in 2004-05 and 23 of the 27 Part B monitoring processes (85%) in 2006-07. 34 In 
three Part B monitoring processes (1 1%) in 2004-05 and four Part B monitoring processes (15%) 
in 2006-07, once the LEA entered the required information into the state’s database, the CAP 
was generated by monitoring software. 

Table 18. Number of state Part B monitoring processes in which CAPs and improvement 
plans were developed by LEAs and/or state agency staff: 2004-05 and 2006-07 



Who developed 


2004-05 


2006-07 


Corrective action plans 1 


32 


32 


LEA staff 


25 


23 


State agency staff 


4 


4 


Both LEA staff and state agency staff 


2 


4 


Improvement plans 2 


23 


21 


LEA staff 


22 


16 


State agency staff 


0 


1 


Both LEA staff and state agency staff 


1 


3 



'Numbers do not sum to totals because for one Part B monitoring process in 2004-05 and for one Part B monitoring 
process in 2006-07, no written CAPs were required. 

2 Numbers do not sum to totals because for one Part B monitoring process in 2006-07, this information was not 
ascertained by the site visit team. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Development of improvement plans . In 2 of the 34 Part B processes in 2004-05 (6%), problems 
were addressed solely through improvement plans; that is, no CAPs were developed. In 2 1 of the 
34 Part B monitoring processes (62%) in 2004-05 and 2 1 of the 32 Part B monitoring processes 
(66%) in 2006-07, though, problems were addressed through improvement plans in addition to 
CAPs. In 15 of these Part B monitoring processes (71%) in 2004-05 and 13 of these Part B 
monitoring processes (62%) in 2006-07, improvement plans were developed separately from 
CAPs. In the remaining six Part B monitoring processes (29%) in 2004-05 and eight Part B 
monitoring processes (38%) in 2006-07, however, corrective action and improvement planning 
were incorporated into a single plan. 



33 In one Part B monitoring process in 2004-05, the state agency did not require written CAPs. 

34 For four Part B monitoring processes in 2004-05, this information was not ascertained by the site visit team. 
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In 15 of the 21 Part B monitoring processes (71%) in 2004-05 and 17 of the 21 Part B 
monitoring processes (81%) in 2006-07 that developed both improvement plans and CAPs, 
improvement plans were developed to address identified problems related to student outcomes or 
performance. In the remaining six Part B monitoring processes (29%) in 2004-05 and four Part B 
monitoring processes (19%) in 2006-07, improvement plans were developed strictly to address 
instances of systemic noncompliance. 

As with CAPs, LEA staff often participated in the development of improvement plans (see table 
18). LEA staff participated in some manner in the development of improvement plans in all 23 of 
the Part B monitoring processes (100%) in 2004-05 and 19 of the 21 Part B monitoring processes 
(90%) in 2006-07 that included an Improvement Planning and Implementation component. In 
one Part B monitoring process (5%) in 2006-07, improvement plans were developed strictly by 

35 

state agency staff. 

As with CAPs, when LEA staff participated in the development of improvement plans (23 of the 
23 Part B monitoring processes in 2004-05 and 19 of the 21 Part B monitoring processes in 
2006-07), the state agency often provided guidance, by providing a specific template for LEAs to 
use, guidelines for the LEAs to follow (e.g., the improvement plan needed to include due dates or 
a section to describe evidence of change), or technical assistance. State agency staff provided 
guidance to LEAs in 12 of the 23 Part B monitoring processes (52%) in 2004-05 and 14 of the 19 
Part B monitoring processes (74%) in 2006-07. 36 

Follow-up to Determine That Plans Were Implemented 

In 8 of the 32 Part B monitoring processes (25%) in 2004-05 and 4 of the 32 (13%) Part B 
monitoring processes in 2006-07 that included a Corrective Action and Enforcement component, 
state agency staff reported that they did not follow up with LEAs to ensure implementation of 
CAPs (see table 19). Likewise, in 8 of the 23 Part B monitoring processes (35%) in 2004-05 and 
3 of the 21 Part B monitoring processes (14%) in 2006-07 that included an Improvement 
Planning and Implementation component, state agency staff reported that they did not follow up 
with LEAs to ensure implementation of improvement plans. 

In the remaining Part B monitoring processes, state agency staff used a variety of approaches to 
assess whether CAPs and improvement plans were implemented (see table 19). This section 
describes those approaches. 

Follow-up on corrective action plans . State agency staff reported following up with LEAs about 
CAPs for 23 of the 32 Part B monitoring processes (72%) in 2004-05 and 28 of the 32 Part B 
monitoring processes (88%) in 2006-07 (see table 19). When following up with LEAs about 
CAPs in 2004-05, the most common approach involved the state initiating communication with 
the LEA to check on progress using phone calls, logs, and e-mails; this approach was used in 10 
Part B monitoring processes (43%) in which there was state follow-up. In 2006-07, the most 
common approach involved having the state monitoring team conduct on-site visits or reviews of 

35 For one Part B monitoring process in 2006-07, this information was not ascertained by the site visit team. 

36 For one Part B monitoring process in 2006-07, this information was not ascertained by the site visit team. 

37 For one Part B monitoring process in 2004-05, this information was not ascertained by the site visit team. 
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data to determine whether corrections had been made; this approach was used in 17 Part B 
monitoring processes (61%) in which there was state follow-up. 

In 3 of the 23 Part B monitoring processes (13%) in 2004-05 and 18 of the 28 Part B monitoring 
processes (64%) in 2006-07, state agency staff combined several approaches to ensure LEA 
follow-through. 

Table 19, Number of state Part B monitoring systems and processes in which various 
follow-up approaches were used to ensure CAP and improvement plan 
implementation: 2004-05 and 2006-07 



Follow-up approach 


2004-05 


2006-07 


Corrective action plans 


32 


32 


General communication (e.g., phone calls, e-mails) 


10 


9 


Submit evidence of implementation 


5 


11 


Progress reports 


2 


11 


Site visits, meetings, data reviews 


9 


17 


None (did not follow up with LEAs) 


8 


4 


Improvement plans 


23 


21 


General communication (e.g., phone calls, e-mails) 


6 


2 


Submit evidence of implementation 


1 


3 


Progress reports 


5 


14 


Site visits, meetings, data reviews 


8 


10 


None (did not follow up with LEAs) 


8 


3 



NOTE: Numbers do not sum to totals because state Part B monitoring processes could use more than one follow-up 
method. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Follow-up on improvement plans . State agency staff reported following up with LEAs about 
improvement plans for 15 of the 23 Part B monitoring processes (65%) in 2004-05 and 17 of the 

TO 

21 Part B monitoring processes (81%) in 2006-07 (see table 19). When following up with 
LEAs about improvement plans in 2004-05, the most common approach involved having the 
state monitoring team conduct on-site visits or reviews of data to detennine whether LEAs were 
making progress on improvement plans; this approach was used in eight Part B monitoring 
processes (53%) in which there was state follow-up. In 2006-07, the most common approach 
involved having LEAs submit progress reports about improvement plan implementation; this 
approach was used in 14 Part B monitoring processes (82%) in which there was state follow-up. 

As with CAPs, in 5 of the 15 Part B monitoring processes (33%) in 2004-05 and 10 of the 17 
Part B monitoring processes (59%) in 2006-07, state agency staff combined several of these 
approaches to ensure LEA follow-through. 



38 For one Part B monitoring process in 2006-07, this information was not ascertained by the site visit team. 
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Reassessment: Part B Monitoring Processes 



According to the framework presented in chapter 3 of this report, Reassessment refers to 
procedures to check to see whether improvement plans or corrective actions have been effective. 
Reassessment is intended to take note of the changes that have occurred following the 
implementation of an improvement plan or a CAP, including whether the original targets are 
now met or at least progress has been made toward achieving those targets. 

As shown in table 8, six Part B monitoring systems (30%) in 2004-05 and five Part B monitoring 
systems (25%) in 2006-07 included procedures for reassessment. Overall, 1 1 of the 34 Part B 
monitoring processes (32%) in 2004-05 and 9 of the 32 Part B processes (28%) in 2006-07 
included a Reassessment component. 

Discussed below is how states’ Part B monitoring processes approached Reassessment, including 
who conducted reassessments and the nature of reassessments. 

Who Conducted Reassessments? 

Reassessments were often conducted by state agency staff (see table 20). Reassessments were 
conducted strictly by state agency staff in 6 of the 11 Part B monitoring processes (55%) in 
2004-05 and 6 of the 9 Part B monitoring processes (67%) in 2006-07. LEA staff, either alone or 
in conjunction with the state agency staff, participated in carrying out reassessments in the 
remaining five Part B monitoring processes (45%) in 2004-05 and two Part B monitoring 
processes (22%) in 2006-07/ When reassessments were conducted by LEAs alone, the findings 
were submitted to the state agency for review. 

Table 20. Number of state Part B monitoring processes in which reassessments were 
conducted by state agency staff and/or LEA staff: 2004-05 and 2006-07 



Who conducted reassessments? 


2004-05 
(/V=l 1) 


2006-07 1 
(N=9) 


LEA staff only 


3 


2 


State agency staff only 


6 


6 


Both LEA staff and state agency staff 


2 


0 



'Numbers do not sum to totals because for one Part B monitoring process in 2006-07, this information was not 
ascertained by the site visit team. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Nature of Reassessments 

In 2004-05, Reassessment was approached in one of two ways. The first approach involved the 
collection and/or analysis of data to determine whether identified problems had been successfully 
addressed by the LEA; this approach was used in 7 of the 1 1 Part B monitoring processes (64%). 
When the identified problems were related to noncompliance, the state monitoring staff 



39 For one of the nine Part B monitoring processes that included a Reassessment component, this information was 
not ascertained by the site visit team. 
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conducted a follow-up on-site visit to review additional student records to determine if the 
identified noncompliance had been addressed. The student records selected during the follow-up 
visit were different from the ones initially reviewed by the state and were ones that had been 
created or modified after the CAPs had been implemented. For example, in one Part B 
monitoring process, the state monitoring team returned to the LEA within 1 year of the initial on- 
site visit and reviewed a new sample of student records, focusing on the previously identified 
areas needing correction. When the identified problems were related to student outcomes or 
performance, data collection/analysis focused on whether the LEA was making progress toward 
meeting the targets for specific indicators. For example, in one Part B monitoring process, LEAs 
were required to develop an improvement plan as a result of an on-site monitoring visit. After a 
period of time, the state agency staff then followed up with the LEA, which involved analyses of 
LEA-level indicator data to detennine whether the improvement strategies were having an 
impact on the LEA performance. 

In 2004-05, the other approach to Reassessment involved requiring LEAs to submit progress 
reports; this approach was used in the remaining four Part B monitoring processes (36%). While 
the exact nature of these reports varied, LEAs were required to submit the reports at least 
annually. In the progress reports, the LEAs typically described the corrective actions or 
improvement strategies that had been implemented and those strategies that resulted in the 
correction of noncompliance or improved perfonnance. For example, in one Part B monitoring 
process, the state agency required its LEAs to complete focused monitoring improvement plans. 
The state monitored implementation of the plans by requiring the LEAs to submit progress 
reports. The LEA progress reports summarized the improvement activities that had been 
implemented and reassessed LEA perfonnance in relation to the target and goals specified in the 
focused monitoring findings. In addition, the state monitoring team verified LEA progress by 
conducting an annual on-site visit that included meeting with the LEA steering committee to 
discuss the LEA’s progress toward meeting the benchmarks and targets included in the focused 
monitoring improvement plan. 

Unlike in 2004-05, Reassessment in 2006-07 was limited to ensuring that noncompliance issues 
had been corrected by the LEAs. As such, Reassessment only involved the collection and/or 
analysis of data to detennine whether identified problems had been successfully addressed by the 
LEA. In 8 of the 9 Part B monitoring processes (89%) that included a Reassessment component, 
additional student records were reviewed to detennine if the identified noncompliance had been 
addressed. 40 The student records selected were different from the ones initially reviewed by the 
state agency or LEA staff. 

Part C Monitoring Processes 

This section presents findings with regard to states’ Part C monitoring processes for each of the 
five framework components. 



40 For one of the nine Part B monitoring processes that included a Reassessment component, this information was 
not ascertained by the site visit team. 
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Table 21 presents the number of Part C monitoring systems and processes in 2004-05 and 2006- 
07 that included each of the components. 41 

Table 21. Number of state Part C monitoring systems and processes that included the 
five framework components: 2004-05 and 2006-07 





2004-05 


2006-07 


Monitoring component 


Systems 

(jV=20) 


Processes 

()V=24) 


Systems 

(A=20) 


Processes 
(N= 28) 


Problem Identification 


20 


24 


20 


28 


Problem Investigation 


6 


8 


6 


8 


Corrective Action and Enforcement 


t 


21 


t 


23 


Improvement Planning and Implementation 


11 


11 


8 


12 


Reassessment 


9 


12 


9 


10 



| Reporting standard not met; exact numbers are not reported to prevent state monitoring systems from being 
identified. 



NOTE: Numbers do not sum to totals because state monitoring systems and processes could include more than one 
component. The data referring to systems present findings for the state monitoring systems overall. States’ 
monitoring systems could be composed of a single set of activities (i.e., a single monitoring process) or multiple sets 
of activities (i.e., multiple monitoring processes). The data referring to processes present findings for the monitoring 
processes that made up the state monitoring systems. If the Part C monitoring system had more than one monitoring 
process, a component was considered present if at least one process included that component. If the Part C 
monitoring system only had one process, then the component had to be part of that process for the component to be 
considered present in the state’s monitoring system. States’ monitoring systems and processes may have included 
additional activities that were not captured by the site visit interviews and ratings. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Table 22 displays how states combined the framework components in their Part C monitoring 
processes. In each time period, the Part C monitoring processes had from one to five 
components, with the Problem Identification component present in all processes. Three of the 24 
Part C monitoring processes (13%) in 2004-05 and 1 of the 28 Part C monitoring processes (4%) 
in 2006-07 included all live framework components. Another seven Part C monitoring processes 
(29%) in 2004-05 and seven Part C monitoring processes (25%) in 2006-07 included four 
components, with the most common combination of components being Problem Identification, 
Corrective Action and Enforcement, Improvement Planning and Implementation, and 
Reassessment. Six Part C monitoring processes (25%) in 2004-05 and 10 Part C monitoring 
processes (36%) in 2006-07 had three components, with the most common combination of 
components in both years being Problem Identification, Corrective Action and Enforcement, and 
Reassessment. Seven additional Part C monitoring processes (29%) in 2004-05 and eight Part C 
monitoring processes (29%) in 2006-07 included two components, either Problem Identification 
and Corrective Action and Enforcement or Problem Identification and Improvement Planning 
and Implementation. The remaining one Part C monitoring process (4%) in 2004-05 and two Part 
C monitoring processes (7%) in 2006-07 included only one component, Problem Identification. 



4i If the state monitoring system had more than one process, a component was considered present in the state’s 
monitoring system if at least one process included that component. If the state monitoring system only had one 
monitoring process, then the component had to be part of that process for the component to be considered present 
in the state’s monitoring system. 
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Table 22. Number of state Part C monitoring processes that included various numbers of 
the five framework components: 2004-05 and 2006-07 



Components 


2004-05 

(#=24) 


2006-07 

(#=28) 


Five components 


3 


1 


Four components 


7 


7 


Problem Identification, Problem Investigation, Corrective Action and 
Enforcement, Improvement Planning and Implementation 


2 


3 


Problem Identification, Problem Investigation, Corrective Action and 
Enforcement, Reassessment 


2 


0 


Problem Identification, Corrective Action and Enforcement, Improvement 
Planning and Implementation, Reassessment 


3 


4 


Three components 


6 


10 


Problem Identification, Problem Investigation, Corrective Action and 
Enforcement 


0 


2 


Problem Identification, Problem Investigation, Improvement Planning and 
Implementation 


1 


2 


Problem Identification, Corrective Action and Enforcement, Improvement 
Planning and Implementation 


1 


1 


Problem Identification, Corrective Action and Enforcement, Reassessment 


4 


5 


Two components 


7 


8 


Problem Identification, Corrective Action and Enforcement 


6 


7 


Problem Identification, Improvement Planning and Implementation 


1 


1 


One component 


1 


2 


Problem Identification 


1 


2 



NOTE: Only those combinations of components that were observed to occur in states’ monitoring processes in 
either 2004-05 or 2006-07 are presented in this table. 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

As with Part B, the remainder of this section presents findings for each of the framework 
components for the Part C monitoring processes. First, a brief summary of the component and 
the number of the Part C monitoring systems and processes that included the component in each 
year (i.e., 2004-05 and 2006-07) is presented. The rest of the discussion is then devoted to 
describing the various ways in which states’ approached that component with respect to their 
Part C monitoring processes. 

Problem Identification: Part C Monitoring Processes 



According to the framework presented in chapter 3 of this report, Problem Identification involves 
determining whether LEAs/EIS programs are in compliance with IDEA regulations and/or are 
performing at satisfactory levels on outcome indicators. Problem Identification includes: (1) 
Indicator and Target Setting, which involves identifying and defining indicators and establishing 
targets for those indicators; (2) Indicator Data Collection and Analysis, which involves collecting, 
preparing, and analyzing data for assessing perfonnance on identified indicators; and (3) Problem 
Detection, which involves comparing perfonnance on a specified indicator to the target established 
for that indicator. 
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As shown in table 21 and noted above, in both 2004-05 and 2006-07, all 20 Part C monitoring 
systems (100%) and all Part C monitoring processes (24 of the 24 Part C monitoring processes in 
2004-05 and 28 of the 28 Part C monitoring processes in 2006-07) included a Problem Identification 
component. 

Discussed below is how states approached Problem Identification in their Part C monitoring 
processes, including how they selected indicators and targets, the data sources they used to identify 
problems, who collected the data, and how the problems were actually identified. 

Selecting Indicators and Targets 

This section discusses how indicators were selected to identify problems, the types of indicators 
that were used (i.e., compliance or outcome indicators), and whether a large set or a small set of 
indicators was chosen. 

Selection of indicators . In 2004-05, in 5 of the 24 Part C monitoring processes (21%), state 
agency staff selected indicators with assistance from a state-level stakeholder committee (see 
table 23). In the remaining 19 Part C monitoring processes (79%), the state agency staff selected 
the indicators to identify problems without input from a state-level stakeholder committee. 

In 2006-07, the state agency staff in 19 of the 28 Part C monitoring processes (68%) selected 
indicators based on the OSEP SPP/APR indicators. Furthermore, for 16 of these 19 Part C 
monitoring processes (84%), state agency staff indicated that they made this decision with input 
from a state-level stakeholder committee. Also, for 14 of the 19 Part C monitoring processes 
(74%), state agency staff noted that they also reviewed state guidelines when selecting their 
indicators and, in some cases, selected indicators in addition to the SPP/APR indicators to 
address specific state requirements. In the remaining nine Part C monitoring processes (32%), 
the state agency staff selected the indicators to identify problems in EIS programs, either with or 
without input from a state-level stakeholder committee. 

Table 23. Number of state Part C monitoring processes that used various approaches to 
select indicators: 2004-05 and 2006-07 



How indicators selected 


2004-05 

(7V=24) 


2006-07 
(N= 28) 


By state, with stakeholder input 


5 


4 


By state, without stakeholder input 


19 


5 


By state, based on State Performance Plan/ Annual 
Perfonnance Plan (SPP/APR) 


t 


19 



(Not applicable. Because the 2004 amendments to IDEA, which required states to submit SPPs/APRs, were not 
enacted until December 2004, this approach was not used by states when selecting their indicators in 2004-05. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Selection of compliance indicators and targets . Overall, 24 of the 24 Part C monitoring processes 
(100%) in 2004-05 and 27 of the 28 Part C monitoring processes (96%) in 2006-07 used 
compliance indicators to assess whether EIS programs were in compliance with the federal 
regulations set forth in IDEA Part C (see table 24). Moreover, 18 of the 24 Part C monitoring 
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processes (75%) in 2004-05 and 12 of the 28 Part C monitoring processes (43%) in 2006-07 used 
only compliance indicators and no outcome indicators. 

The targets for compliance indicators were generally 0 or 100 percent, as required by law. As 
with Part B, however, 6 of the 24 Part C monitoring processes (25%) in 2004-05 and 6 of the 27 
Part C monitoring processes (22%) in 2006-07 that used compliance indicators either set targets 
below 100 percent or determined noncompliance subjectively. For example, in one Part C 
monitoring process, state agency staff set levels of acceptable perfonnance where 90 percent or 
greater indicated compliance, between 80 and 89 percent indicated partial noncompliance, and 
less than 80 percent indicated noncompliance. In another Part C monitoring process, there were 
no set targets for noncompliance; instead, this determination was left up to the judgment of the 
state agency staff. 

Table 24. Number of state Part C monitoring processes that used compliance and 
outcome indicators: 2004-05 and 2006-07 



Indicator types and combinations 


2004-05 

(jV=24) 


2006-07 

(A=28) 


Indicator types 


Compliance 


24 


27 


Outcome 


6 


16 


Indicator combinations 


Compliance only 


18 


12 


Outcome only 


0 


1 


Both compliance and outcome 


6 


15 



NOTE: Numbers do not sum to totals for indicator types because state monitoring processes could use more than 
one type of indicator. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Selection of outcome indicators and targets . Six of the 24 Part C monitoring processes (25%) in 
2004-05 and 16 of the 28 Part C monitoring processes (57%) in 2006-07 used outcome indicators 
(see table 24). 

State agency staff in 5 of the 6 Part C monitoring processes (83%) in 2004-05 and 14 of the 16 
Part C monitoring processes (88%) in 2006-07 that included outcome indicators set targets for 
these indicators. For those Part C monitoring processes in which targets were not set (1 of the 6 
Part C monitoring processes or 17% in 2004-05 and 2 of the 16 Part C monitoring processes or 
13% in 2006-07), the state agency staff either ranked the EIS programs and used state-defined 
criteria to select the lowest ranking ones, or performance was subjectively determined by the 
state agency. 

Number of indicators used . As shown in table 25, in 21 of the 24 Part C monitoring processes 
(88%) in 2004-05 and 22 of the 28 Part C monitoring processes (79%) in 2006-07, state agency 
staff used a large number of compliance and/or outcome indicators to identify problems in EIS 
programs (i.e., anywhere from a dozen or so to hundreds in some cases). The remaining three 
Part C monitoring processes (13%) in 2004-05 and six Part C monitoring processes (21%) in 
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2006-07 focused on a small set of indicators (i.e., between one and four indicators or priority 
areas). 

Table 25. Number of state Part C monitoring processes that used either a large or small 
set of compliance and/or outcome indicators: 2004-05 and 2006-07 





2004-05 


2006-07 


Number of indicators used 


{N= 24) 


(7V= 28) 


Small set of indicators 


3 


6 


Large set of indicators 


21 


22 



NOTE: A small set of indicators was defined as between one and four indicators or priority areas; a large set of 
indicators was defined as five or more indicators or priority areas. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Problem Identification Data Sources and Collection 

This section describes the data sources used to assess local performance on the indicators and 
who typically was involved in the data collections, the state agency staff, the EIS staff, or both. 

Problem Identification data sources . A number of data sources were used to identify problems 
(see table 26). The most common Part C data source was record review, which was used in 20 of 
the 24 Part C monitoring processes (83%) in 2004-05 and 22 of the 28 Part C monitoring 
processes (79%) in 2006-07. State agency staff also reported that they used existing state or local 
data; this data source was used in 14 of the 24 Part C monitoring processes (58%) in 2004-05 and 
19 of the 28 Part C monitoring processes (68%) in 2006-07. Stakeholder interviews or 
forums/focus groups were used in 10 of the 24 Part C monitoring processes (42%) in 2004-05 
and 9 of the 28 Part C monitoring processes (32%) in 2006-07. Other data sources included 
surveys (often parent surveys) and observations of children. 

Table 26. Number of state Part C monitoring processes that used various data sources to 
identify problems: 2004-05 and 2006-07 



Data source 


2004-05 

(A=24) 


2006-07 
(N= 28) 


Record reviews 


20 


22 


Interviews/forums/focus groups 


10 


9 


Existing state/local data 


14 


19 


Surveys 


6 


9 


Child observations 


2 


2 



NOTE: Numbers do not sum to totals because state monitoring processes could use more than one data source. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

A number of Part C monitoring systems (15 of the 24 Part C monitoring processes or 63% in 
2004-05 and 17 of the 28 Part C monitoring processes or 61% in 2006-07) combined two or 
more of these sources of data in order to identify problems in EIS programs. For example, state 
agency staff for one Part C monitoring process required its EIS programs to complete a self- 
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assessment, which included analyzing 618 data, conducting record reviews, and administering a 
parent survey. EIS programs then used all of this information to identify problems. Nine of the 
24 Part C monitoring processes (38%) in 2004-05 and 1 1 of the 28 Part C monitoring processes 
(39%) in 2006-07 used only one of these sources of data to identify problems in EIS programs. 
For example, in one Part C monitoring process, state agency staff reported that they identified 
problems in EIS programs solely through the use of on-site record reviews. 

Who collected data for Problem Identification ? As shown in table 27, EIS program staff, either 
alone or in conjunction with the state agency staff, participated in the data collection in 15 of the 
24 Part C monitoring processes (63%) in 2004-05 and 20 of the 28 Part C monitoring processes 
(71%) in 2006-07. In the remaining nine Part C monitoring processes (38%) in 2004-05 and 
eight Part C monitoring processes (29%) in 2006-07; however, data collection was completed 
solely by the state agency staff, either alone or with assistance from others, such as consultants, 
parents, or local administrators from EIS programs other than the one being monitored. 

Table 27. Number of state Part C monitoring processes in which data for identifying 
problems were collected by EIS program staff and/or state agency staff: 2004- 
OS and 2006-07 



Who collected the data? 


2004-05 

(A=24) 


2006-07 
(N= 28) 


EIS program staff 


7 


11 


State agency staff 


9 


8 


Both EIS program staff and state agency staff 


8 


9 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Identifying Problems 

This section describes how problems were identified for compliance and outcome indicators and 
who typically identified the problems, either the state agency staff, the EIS program, or both. 

Identifying problems for compliance indicators . In 16 of the 24 Part C monitoring processes 
(67%) in 2004-05 and 18 of the 27 Part C monitoring processes (67%) in 2006-07 that included 
compliance indicators, identifying problems with regard to compliance-based indicators involved 
comparing the EIS program’s perfonnance on the compliance indicator to the target of 100 
percent. If the EIS program’s perfonnance was less than the 100 percent target, then a problem 
was identified. As discussed previously, however, state agency staff in 6 of the 24 Part C 
monitoring processes (25%) in 2004-05 and 6 of the 27 Part C monitoring processes (22%) in 
2006-07 set targets that were less than 100 percent, and compliance issues were identified only 
when a certain percentage of the child records reviewed were found to be in noncompliance on 
the same indicator, or compliance problems were determined subjectively. Also, in 2 of the 24 
Part C monitoring processes (8%) in 2004-05 and 3 of the 27 Part C monitoring processes (11%) 
in 2006-07, instead of using the targets for the compliance indicators to identify problems, the 
state agency staff ranked the EIS programs based on their performance on the indicators and then 
used state-defined criteria to select the bottom-ranking programs. 
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Identifying problems for outcome indicators . In 2004-05, problem identification for five of the 
six Part C monitoring processes (83%) that had outcome indicators involved comparison of the 
EIS program performance to the targets that were set for those indicators. Problems were 
identified when performance did not meet or exceed the target. In the one remaining Part C 
monitoring process (17%), it was up to the EIS program to identify problems using its own 
standards, which could include setting its own targets or comparing its performance to all the EIS 
programs in the state or to EIS programs that were similar in nature. 

In 2006-07, problems were typically identified using one of two methods: (1) the EIS program’s 
performance was compared to the target that had been set for that indicator, or (2) the EIS 
programs were ranked in some manner, and the lowest performing EIS programs on the 
indicators were selected for further monitoring activities. Overall, 12 of the 16 Part C monitoring 
processes (75%) that used outcome indicators incorporated the first method of identifying 
problems, while the 4 remaining Part C monitoring processes (25%) incorporated the second 
method. 

Who identified problems ? In 23 out of the 24 Part C monitoring processes (96%) in 2004-05 and 
23 of the 28 Part C monitoring processes (82%) in 2006-07, the state agency staff were involved 
in identifying problems (see table 28). In 20 of the 24 Part C monitoring processes (83%) in 
2004-05 and 22 of the 28 Part C monitoring processes (79%) in 2006-07, all problems were 
identified by the state agency. In 3 of the 24 Part C monitoring processes (13%) in 2004-05 and 1 
of the 28 Part C monitoring processes (4%) in 2006-07, the state agency staff and the EIS 
program staff jointly identified problems. For the remaining Part C monitoring process (4%) in 
2004-05 and five Part C monitoring processes (18%) in 2006-07, all problems were identified by 
the EIS programs. 

Table 28. Number of state Part C monitoring processes in which problems were 
identified by state agency staff and/or EIS program staff: 2004-05 and 2006-07 



Who identified problems? 


2004-05 
(N= 24) 


2006-07 
(N= 28) 


EIS program staff 


1 


5 


State agency staff 


20 


22 


Both EIS program staff and state agency staff 


3 


1 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Problem Investigation: Part C Monitoring Processes 



The previous section discussed state approaches to identifying problems. According to the 
framework presented in chapter 3 of this report, once a problem is identified, Problem 
Investigation is needed to gather additional information or data to explain why the problem exists 
and what needs to be done to address it. 
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As shown in table 21, six Part C monitoring systems (30%) in both 2004-05 and 2006-07 
included procedures for investigating problems that were identified in EIS programs. Overall, 8 
of the 24 Part C monitoring processes (33%) in 2004-05 and 8 of the 28 Part C monitoring 
processes (29%) in 2006-07 included a Problem Investigation component. 

Discussed below is how states approached Problem Investigation in their Part C monitoring 
processes, including whether problem investigations were conducted by state agency staff or EIS 
program staff, the nature of problem investigations, and the kinds of data and information used to 
investigate problems. 

Who Conducted Problem Investigations? 

Problem investigations were carried out solely by state agency staff (or groups contracted by the 
state) in six of the eight Part C monitoring processes (75%) in 2004-05 and five of the eight Part 
C monitoring processes (63%) in 2006-07, often as part of on-site visits (see table 29). In two of 
the eight Part C monitoring processes (25%) in 2004-05 and one of the eight Part C monitoring 
processes (13%) in 2006-07, the state agency staff and the EIS program staff jointly conducted 
problem investigations. In the remaining two Part C monitoring processes (25%) in 2006-07, EIS 
program staff conducted problem investigations, with guidance from the state agency. 

Table 29. Number of state Part C monitoring processes in which problems were 
investigated by state agency staff and/or EIS program staff: 2004-05 and 2006- 
07 



Who investigated problems? 


2004-05 
(N= 8) 


2006-07 

(A=8) 


EIS program staff 


0 


2 


State agency staff 


6 


5 


Both EIS program staff and state agency staff 


2 


1 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Tailoring of Problem Investigations 

In seven of the eight Part C monitoring processes (88%) in 2004-05 and five of the eight Part C 
monitoring processes (63%) in 2006-07, the same approach to problem investigation was used 
across all EIS programs. For example, in one Part C monitoring process, once a problem was 
identified in an EIS program, the state agency staff engaged in a review of its policies and 
procedures in order to try to detennine the cause of the problem. This approach was used for all 
EIS programs regardless of the specific problems that had been identified. In the remaining Part 
C monitoring process (13%) in 2004-05 and three Part C monitoring processes (38%) in 2006- 
07, the exact approach or approaches were based on the nature of the problem under 
investigation. For example, in one Part C monitoring process, the state monitoring teams tailored 
their approach to explore why a particular problem existed in an EIS program. Once an EIS 
program had been identified, the team met to review additional data and, based on the findings, 
developed hypotheses that then guided the on-site visit. While on-site, the team could use record 
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reviews, stakeholder interviews, and child observations to investigate the problem, all of which 
were tailored to fit the team’s hypotheses. 

Data and Information Used to Investigate Problems 

This section describes the data sources and information that were used to investigate problems in 
EIS programs. 

Previously collected data or information . In all eight of the Part C monitoring processes (100%) 
in 2004-05 and six of the eight Part C monitoring processes (75%) in 2006-07 that included a 
Problem Investigation component, the investigation of problems included the review of data or 
information that had been previously collected for other purposes (see table 30). The types of 
data and/or infonnation reviewed included previous monitoring reports and findings, self- 
assessment findings, hearing/mediation data, compliance data, and data collected for section 618 
of IDEA. For example, in one Part C monitoring process, problem investigation was conducted 
as part of focused monitoring. Once an EIS program was identified as having poor perfonnance, 
the state monitoring staff conducted an on-site visit to further investigate and to try to determine 
possible causes of the poor performance. The on-site activities included reviews of existing 
compliance, hearing/mediation, and outcome data. 

Table 30. Number of state Part C monitoring processes that used previously collected 
data and newly collected data to investigate problems: 2004-05 and 2006-07 



Data source 


2004-05 

(N= 8) 


2006-07 
(N= 8) 


Previously collected data or information 


8 


6 


Newly collected data 


5 


5 


Record review data 


5 


4 


Interviews/forums/focus groups 


2 


4 


Child observations 


0 


2 



NOTE: Numbers do not sum to totals because state monitoring processes could use more than one data source. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Newly collected data . In five of the eight Part C monitoring processes (63%) in both 2004-05 
and 2006-07, problem investigations involved the collection and analysis of new data — mostly 
qualitative data — from multiple sources (see table 30). In all five of the Part C monitoring 
processes (100%) in 2004-05 and four of the five Part C monitoring processes (80%) in 2006-07 
that collected new data, state agency staff reported collecting data through reviews of child 
records. Child records were either selected purposefully, based on the problem(s) being 
investigated, or selected randomly. Conducting interviews and/or forums and focus groups to 
collect additional information about identified problems was used in two of the five Part C 
monitoring processes (40%) in 2004-05 and four of the five Part C monitoring processes (80%) 
in 2006-07. State agency staff reported gathering information from local Part C providers, 
parents, local interagency coordinating council members, and agency partners. In 2006-07, state 
agency staff in two of the five Part C monitoring processes (40%) conducted child observations; 
state agency staff did not report conducting child observations in 2004-05. 
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Addressing Identified Problems: Part C Monitoring Processes 



According to the framework presented in chapter 3 of this report, once a state identifies problems 
and conducts an investigation to explore why those problems exist, the next step is to address 
those problems. Two components from the framework are related to addressing identified 
problems: (1) Corrective Action and Enforcement, which is used when identified problems are 
related to noncompliance with procedural requirements or requirements to provide appropriate 
services in a timely manner, 4 " and (2) Improvement Planning and Implementation, which is 
typically used when identified problems are more systemic (either locally or statewide), such as 
when there is pervasive noncompliance in certain areas or the problems involve 
underperformance on selected targets or benchmarks. 

In 2004-05, all 20 of the Part C monitoring systems (100%) included procedures to address the 
problem(s) identified through their monitoring activities. In 2004-05, a total of 9 of the 20 Part C 
monitoring systems (45%) addressed identified problems through both Corrective Action and 
Enforcement and Improvement Planning and Implementation. The remaining 1 1 Part C 
monitoring systems (55%) addressed identified problems either solely through Corrective Action 
and Enforcement or solely through Improvement Planning and Implementation. 43 Overall, 21 out 
of the 24 Part C monitoring processes (88%) in 2004-05 included Corrective Action and 
Enforcement, and 1 1 Part C monitoring processes (46%) included Improvement Planning and 
Implementation (see table 21). 

In 2006-07, most Part C monitoring systems included procedures to address the problem(s) 
identified through their Part C monitoring activities. 44 In 2006-07, seven of these Part C 
monitoring systems (35%) addressed identified problems through both Corrective Action and 
Enforcement and Improvement Planning and Implementation. The remaining Part C monitoring 
systems addressed identified problems solely through Corrective Action and Enforcement or 
solely through Improvement Planning and Implementation. 4 ' 2 Overall, 23 out of the 28 Part C 
monitoring processes (82%) in 2006-07 included Corrective Action and Enforcement, and 12 of 
the 28 Part C monitoring processes (43%) included Improvement Planning and Implementation. 

Discussed below is how the problems identified in EIS programs were addressed by states’ Part 
C monitoring processes. In general, two activities were included: (1) the development of a plan 
outlining the identified problems that were serious enough to require some action by the LEA 
and (2) some type of follow-up to determine that the plan was implemented. 

Development of a Plan 

Generally, a written plan was required in response to findings of noncompliance or other 
identified problems. This section describes the development and nature of CAPs and 
improvement plans. 



42 As noted in the framework, “corrective action” has a specific meaning in IDEA. Findings of noncompliance must 
be corrected as soon as possible, and in no case later than 1 year after identification. 

43 See footnote 13 on page 41. 

44 See footnote 13 on page 41. 

45 See footnote 13 on page 41. 
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Development of corrective action plans . CAPs were developed to address problems related to 
noncompliance. In 5 of the 21 Part C monitoring processes (24%) in 2004-05 and 1 of the 23 
Part C monitoring processes (4%) in 2006-07 that included a Corrective Action and Enforcement 
component, CAPs were developed strictly to address instances of noncompliance with 
procedures and timelines at the individual child level. In these Part C monitoring processes, 
systemic noncompliance was addressed through improvement plans. In the majority of Part C 
monitoring processes, however, CAPs were developed to address both individual child-level and 
systemic noncompliance (16 of the 21 Part C monitoring processes or 76% in 2004-05 and 22 of 
the 23 Part C monitoring processes or 96% in 2006-07). 

EIS program staff typically developed CAPs (see table 31). In 13 of the 21 Part C monitoring 
processes (62%) in 2004-05 and 15 of the 23 Part C monitoring processes (65%) in 2006-07, the 
EIS program staff were primarily responsible for developing CAPs. In an additional five Part C 
monitoring processes in both 2004-05 and 2006-07 (24% and 22%, respectively), EIS program 
staff and state agency staff jointly developed CAPs. For example, one Part C monitoring process 
operated on a 3-year cycle that was supplemented by electronic file reviews. Following an on- 
site visit by a state team and preparation of written findings, state personnel and the EIS program 
together developed an action plan for areas that were out of compliance or needed attention. In 
the remaining two Part C monitoring processes (10%) in 2004-05 and three Part C monitoring 
processes (13%) in 2006-07, the state agency staff developed the CAPs. 46 

Table 31. Number of state Part C monitoring processes in which CAPs and improvement 
plans were developed by EIS program and/or state agency staff: 2004-05 and 
2006-07 



Who developed CAPs or improvement plans? 


2004-05 1 


2006-07 


Corrective action plans 1 


21 


23 


EIS program staff 


13 


15 


State agency staff 


2 


3 


Both EIS program staff and state agency staff 


5 


5 


Improvement plans 


11 


12 


EIS program staff 


8 


10 


State agency staff 


0 


1 


Both EIS program staff and state agency staff 

1-VT1 1 i iiill r* T\ i /~t ’ i ’ 


3 


1 



'Numbers do not sum to totals because for one Part C monitoring process in 2004-05, written CAPs were not 



required. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

When EIS program staff participated in the development of CAPs (18 Part C monitoring 
processes in 2004-05 and 20 Part C monitoring processes in 2006-07), state agency staff often 
provided guidance, either by providing a specific template for EIS program staff to use, 
guidelines for the EIS program to follow (e.g., the CAP needed to include steps to be taken or a 
section to describe evidence of correction) or technical assistance. State agency staff provided 



46 For one Part C monitoring process in 2004-05, written CAPs were not required. 
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guidance to EIS programs in 17 of the 18 Part C monitoring processes (94%) in 2004-05 and 17 
of the 20 Part C monitoring processes (85%) in 2006-07. 

Development of improvement plans . Two of the 1 1 Part C monitoring processes (18%) in 2004- 
05 and 3 of the 12 Part C monitoring processes (25%) in 2006-07 that included an Improvement 
Planning and Implementation component addressed problems through improvement plans only; 
that is, no CAPs were developed. 

In the remaining nine Part C monitoring processes in both 2004-05 and 2006-07 (82% and 75%, 
respectively), problems were addressed through improvement plans in addition to CAPs. In three 
of these nine Part C monitoring processes (33%) in 2004-05 and seven of these nine Part C 
monitoring processes (78%) in 2006-07, improvement plans were developed separately from 
CAPs. In the remaining six Part C monitoring processes (67%) in 2004-05 and two Part C 
monitoring processes (22%) in 2006-07, however, corrective action and improvement planning 
were incorporated into a single plan. For example, in one Part C monitoring process, EIS 
programs were required to develop an action plan that responded to written findings from the 
state agency’s on-site review. The plan needed to address all identified problem areas — which 
the state agency divided into “short/simple” and “intensive/longer-term” problems. Thus, in this 
Part C monitoring process, the action plan combined corrective action and improvement 
activities. 

In five of the nine Part C monitoring processes (56%) in 2004-05 and one of the nine Part C 
monitoring processes (11%) in 2006-07 that used both CAPs and improvement plans to address 
problems, improvement plans were developed strictly to address instances of systemic 
noncompliance. In the remaining four Part C monitoring processes (44%) in 2004-05 and eight 
Part C monitoring processes (88%) in 2006-07, improvement plans were developed to address 
identified problems related to child outcomes or perfonnance. 

As with CAPs, EIS program staff often participated in the development of improvement plans 
(see table 31). EIS program staff participated in some manner in the development of 
improvement plans in all 11 Part C monitoring processes (100%) in 2004-05 and 11 of the 12 
Part C monitoring processes (92%) in 2006-07 that included an Improvement Planning and 
Implementation component. In one Part C monitoring process (8%) in 2006-07, improvement 
plans were developed strictly by state agency staff. 

When EIS program staff developed improvement plans (11 Part C monitoring processes in both 
2004-05 and 2006-07), the state agency often provided guidance, by providing a specific 
template, guidelines for the EIS programs to follow (e.g., the improvement plan needed to 
include due dates or a section to describe evidence of change), or technical assistance. State 
agency staff provided guidance to EIS programs in 9 of the 1 1 Part C monitoring processes 
(82%) in both 2004-05 and 2006-07. 47 

Follow-up to Determine That Plans Were Implemented 

In 3 of the 21 Part C monitoring processes (14%) in 2004-05 and 1 of the 23 Part C monitoring 
processes (4%) in 2006-07 that included a Corrective Action and Enforcement component, state 

47 For one Part C monitoring process in 2004-05, this information was not ascertained by the site visit team. 
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agency staff reported that they did not follow up with EIS programs to ensure implementation of 
CAPs (see table 32). Likewise, in 3 of 1 1 Part C monitoring processes (27%) in 2004-05 and 2 of 
the 12 Part C monitoring processes (17%) in 2006-07 that included an Improvement Planning 
and Implementation component, state agency staff reported that they did not follow up with EIS 
programs to ensure implementation of improvement plans. 

In the remaining Part C monitoring processes, state agency staff used a variety of approaches to 
assess whether CAPs and improvement plans were implemented (see table 32). This section 
describes those approaches. 

Table 32. Number of state Part C monitoring processes in which various follow-up 
methods were used to ensure CAP and improvement plan implementation: 
2004-05 and 2006-07 



Follow-up approach 


2004-05 


2006-07 


Corrective action plans 


21 


23 


General communication (e.g., phone calls, e-mails) 


4 


1 


Submit evidence of implementation 


1 


9 


Progress reports 


10 


6 


Site visits, meetings, data reviews 


11 


14 


None (did not follow up with EIS programs) 


3 


1 


Improvement plans 


11 


12 


General communication (e.g., phone calls, e-mails) 


3 


3 


Submit evidence of implementation 


0 


2 


Progress reports 


6 


7 


Site visits, meetings, data reviews 


6 


6 


None (did not follow up with EIS programs) 


3 


2 



NOTE: Numbers do not sum to totals because Part C monitoring processes could use more than one follow-up 
method. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Follow-up on CAPs . State agency staff reported following up with EIS programs about CAPs in 
18 of the 21 Part C monitoring processes (86%) in 2004-05 and 22 of the 23 Part C monitoring 
processes (96%) in 2006-07 that included a Corrective Action and Enforcement component (see 
table 32). When following up with EIS programs about CAPs, the most common approach in 
both 2004-05 and 2006-07 involved having the state monitoring team conduct on-site visits or 
reviews of data to determine whether corrections had been made; this approach was used in 1 1 of 
the 18 Part C monitoring processes (61%) in 2004-05 and 14 of the 22 Part C monitoring 
processes (64%) in 2006-07 in which there was state follow-up. In 2004-05, another common 
approach involved having the EIS program submit progress reports to the state agency about 
CAP implementation; this approach was used by 10 of the 18 Part C monitoring processes (56%) 
in which there was state follow-up. Another common approach that was used in 2006-07 
involved having the EIS programs submit evidence to the state agency that they had made all 
needed corrections; this approach was used in 9 of the 22 Part C monitoring processes (41%) in 
which there was state follow-up. 
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In some cases, state agency staff combined several approaches to ensure EIS program follow- 
through (6 of the 18 Part C monitoring processes or 33% in 2004-05 and 8 of the 22 Part C 
monitoring processes or 36% in 2006-07). 

Follow-up on improvement plans . State agency staff reported following up with EIS programs 
about improvement plans in 8 of the 1 1 Part C monitoring processes (73%) in 2004-05 and 10 of 
the 12 Part C monitoring processes (83%) in 2006-07 that included an Improvement Planning 
and Implementation component (see table 32). When following up with EIS programs about 
improvement plans, the most common approaches in both 2004-05 and 2006-07 involved (1) 
having the state monitoring team conduct on-site visits or reviews of data to detennine whether 
EIS programs were making progress on improvement plans and (2) having EIS programs submit 
progress reports about improvement plan implementation. Six of the eight Part C monitoring 
processes (75%) in 2004-05 and 6 of the 10 Part C monitoring processes (60%) in 2006-07 used 
the first approach (i.e., site visits and data reviews) and six of the eight (75%) Part C monitoring 
processes in 2004-05 and 7 of the 10 Part C monitoring processes (70%) in 2006-07 used the 
second approach (i.e., progress reports). 

As with CAPs, in some cases, state agency staff combined several approaches to ensure EIS 
program follow-through (5 of the 8 Part C monitoring processes or 63% in 2004-05 and 5 of the 
10 Part C monitoring processes or 50% in 2006-07). 

Reassessment: Part C Monitoring Processes 

According to the framework presented in chapter 3 of this report, Reassessment refers to 
procedures to check to see whether improvement plans or corrective actions have been effective. 
Reassessment is intended to take note of the changes that have occurred following the 
implementation of an improvement plan or a CAP, including whether the original targets are 
now met, or at least moving in a direction that demonstrates progress. 

As shown in table 21, nine Part C monitoring systems (45%) in both 2004-05 and 2006-07 
included procedures for Reassessment. Overall, 12 of the 24 Part C monitoring processes (50%) 
in 2004-05 and 10 of the 28 Part C processes (36%) in 2006-07 included a Reassessment 
component. 

Discussed below is how states’ Part C monitoring processes approached Reassessment, including 
who conducted reassessments and the nature of reassessments. 

Who Conducted Reassessments? 

In 5 of the 12 Part C monitoring processes (42%) in 2004-05 and 7 of the 10 Part C monitoring 
processes (71%) in 2006-07 that conducted reassessments, these activities were completed solely 
by the state agency staff (see table 33). In 4 of the 12 Part C monitoring processes (33%) in 
2004-05 and 1 of the 10 Part C monitoring processes (10%) in 2006-07, reassessments were 
conducted by state agency staff and EIS program staff jointly. In the remaining three Part C 
monitoring processes (25%) in 2004-05 and one Part C monitoring processes (14%) in 2006-07, 
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reassessments were conducted by EIS program staff, who then provided updates to the state 

48 

agency. 

Table 33. Number of state Part C monitoring processes in which reassessments are 
conducted by state agency staff and/or EIS staff: 2004-05 and 2006-07 



Who conducted reassessments? 


2004-05 
(N= 12) 


2006-07 1 

(A=10) 


EIS program staff 


3 


1 


State agency staff 


5 


7 


Both EIS program staff and state agency staff 


4 


1 



'Numbers do not sum to totals because for one Part C monitoring process in 2006-07, this information was not 
ascertained by the site visit team. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Nature of Reassessments 

In 2004-05, Reassessment was approached in two ways in Part C monitoring processes; these 
two approaches were used either alone or in conjunction with one another. The first approach to 
Reassessment involved the analysis of new or existing data (9 of the 12 Part C monitoring 
processes or 75%), such as record reviews or desk audits. For example, in one Part C monitoring 
process, if a noncompliance finding was determined to be serious or systemic, a follow-up 
review was conducted approximately 6 months following the approval of the CAP. The follow- 
up visit included pulling new records for review, focusing only on reassessment of perfonnance 
in the area(s) of noncompliance to determine whether the CAP had successfully addressed those 
problems. The second approach to Reassessment that was used in Part C monitoring processes 
included progress reports submitted by the EIS program to the state agency; this approach was 
used in six of the seven Part C monitoring processes (86%) in which EIS program staff were 
involved in reassessments. The state agency staff required the EIS programs to submit these 
reports so that they could stay abreast of the EIS program’s progress toward addressing identified 
problems. 

In three Part C monitoring processes, state agency staff combined these two approaches (i.e., the 
analysis of new or existing data and the submission of progress reports). For example, in one Part 
C monitoring process, the state agency staff conducted monthly desk audits and required that the 
EIS program submit monthly reports, both of which included new data to reassess the EIS 
program’s performance. This was done until the EIS program could prove 100 percent 
compliance had been achieved. 

In 2006-07, reassessments for Part C in 2006-07 were limited to ensuring that noncompliance 
issues had been corrected by the EIS programs. As such, Reassessment primarily involved the 
collection and/or analysis of data to determine whether identified problems had been successfully 
addressed by the EIS programs. In 9 of the 10 Part C monitoring processes (90%), additional 



48 For one Part C monitoring process in 2006-07, this information was not ascertained by the site visit team. 



86 




child records were reviewed to determine if the identified noncompliance had been addressed. 49 
The child records selected were different from the ones initially reviewed by the state or EIS 
program. In one of the two Part C monitoring processes (50%) in which the EIS program staff 
were involved in reassessments, the EIS program was also required to submit progress reports to 
the state agency. 

Summary for Mapping Onto the Framework Components 

This section focused on how states’ Part B and Part C monitoring processes mapped onto the 
framework components. Few Part B and Part C monitoring processes included all live of the 
framework components in either 2004-05 (4 of the 34 Part B monitoring processes and 3 of the 
24 Part C monitoring processes) or 2006-07 (4 of the 32 Part B monitoring processes and 1 of the 
28 Part C monitoring processes). For Part B, states’ monitoring processes most commonly 
included three components in 2004-05 (14 of the 34 Part B monitoring processes) and four 
components in 2006-07 (14 of the 32 Part B monitoring processes). For Part C, in 2004-05, equal 
numbers of monitoring processes included either two components (7 of the 24 Part C monitoring 
processes) or four components (7 of the 24 Part C monitoring processes); in 2006-07, states’ 
monitoring processes most commonly included three components (10 of the 24 Part C 
monitoring processes). 

In both 2004-05 and 2006-07, all Part B and Part C monitoring processes included Problem 
Identification, and most included Corrective Action and Enforcement (32 of the 34 Part B 
monitoring processes in 2004-05 and all 32 Part B monitoring processes in 2006-07; 21 of the 24 
Part C monitoring processes in 2004-05 and 23 of 28 the Part C monitoring processes in 2006- 
07). For Part B, 16 of the 34 monitoring processes in 2004-05 and 17 of the 32 monitoring 
processes in 2006-07 included Problem Investigation; 23 of the 34 monitoring processes in 2004- 
05 and 21 of the 32 monitoring processes in 2006-07 included Improvement Planning and 
Implementation; and 1 1 of the 34 monitoring processes in 2004-05 and 9 of the 32 monitoring 
processes in 2006-07 included Reassessment. For Part C, 8 of the 24 monitoring processes in 
2004-05 and 8 of the 28 monitoring processes in 2006-07 included Problem Investigation; 1 1 of 
the 24 monitoring processes in 2004-05 and 12 of the 28 monitoring processes in 2006-07 
included Improvement Planning and Implementation; and 12 of the 24 monitoring processes in 
2004-05 and 10 of the 28 monitoring processes in 2006-07 included Reassessment. 



Analysis of Framework Components and Elements 

The framework was developed to describe the nature and design of state monitoring systems and 
processes. Each component of the framework includes key characteristics that identify aspects 
that are thought to be important to the component being carried out. The elements then spell out 
what signifies the presence of each key characteristic in states’ Part B and Part C monitoring 
systems and processes. 



49 For one Part C monitoring process in 2006-07, this information was not ascertained by the site visit team. 
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A specific purpose of the site visits to the 20 randomly selected states was to assess the extent to 
which elements associated with the framework components were present in each state’s Part B 
and Part C monitoring processes. This section first describes findings related to the component 
scores that were calculated for all Part B and Part C monitoring processes. Then it explores the 
data in greater detail by reviewing which of the individual elements associated with each 
component of the framework were present in states’ Part B and Part C monitoring processes. 
Data on the key characteristics are not presented due to the small number of elements associated 
with some key characteristics, which affects the reliability of scores. 

A couple of limitations are associated with this analysis and should be kept in mind. The first 
limitation concerns the framework itself. The logic of the framework and its components 
represent the work of the study team with input from the Advisory Panel and OSEP. As noted 
earlier, the framework has not been evaluated or validated. Second, the site visit ratings on which 
these analyses are based focus primarily on the presence or absence of particular elements in 
state monitoring processes rather than the quality of those elements. That is, “Was the element 
present?” as opposed to “How good is the element?” Thus, for the Problem Investigation element 
that reads “Procedures are in place to monitor the quality of data collected” (see appendix B, 
D5A), site visitors rated whether there were such procedures, not how good they were. 

Average Component Scores for Part B and Part C Monitoring Processes 

Component scores were calculated for each of the Part B and Part C monitoring processes (see 
chapter 4 and appendix H for more information about the creation of the component scores). In 
creating the component scores, IRT modeling was used rather than simply adding up the total 
number of elements present and calculating percentages. This allowed the elements to be 
weighted differently based on their specific properties (e.g., the frequency with which particular 
elements were rated as present or absent). It also made it possible to equate the two sets of ratings 
since different length rating forms were used during the two rounds of site visits. This was done so 
that the component scores from the first round of site visits would have the same meaning as the 
component scores from the second round of site visits. 

The resulting component scores represent the estimated percentage of elements for a particular 
component that were present in a given Part B or Part C monitoring process. So, for example, if a 
process received a component score of 58.0 for the Problem Investigation component, it means 
that an estimated 58.0 percent of the elements for this component were present for that process. 

Average component scores were then calculated for Problem Identification, Problem 
Investigation, Corrective Action and Enforcement, Improvement Planning and Implementation, 
and Reassessment. This section reports these average component scores, first for the Part B 
monitoring processes for both 2004-05 and 2006-07 and then for the Part C monitoring processes 
for both 2004-05 and 2006-07. As discussed in the previous section, not all Part B and Part C 
monitoring processes included all five of the framework components. The resulting average 
component scores can be interpreted as the average estimated percentage of elements present for 
those monitoring processes that included that component. For example, if 10 monitoring 
processes included a Reassessment component and the average component score was 17.0, it 
means that for those 10 processes, on average, 17.0 percent of the Reassessment elements were 
present. 
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Part B Average Component Scores 



There were a total of 34 Part B monitoring processes in 2004-05 and 32 Part B monitoring 
processes in 2006-07. All of the Part B monitoring processes in 2004-05 and 2006-07 included a 
Problem Identification component (see table 34). The average estimated percentage of the 
Problem Identification elements present was 58.6 in 2004-05 and 71.4 in 2006-07. 
Approximately one -half of the Part B monitoring processes in 2004-05 (16 of the 34 processes) 
and 2006-07 (17 of the 32 processes) included a Problem Investigation component. For those 
processes that included the component, the average estimated percentage of the Problem 
Investigation elements present was 72.1 in 2004-05 and 71.9 in 2006-07. Most of the Part B 
monitoring processes (32 of the 34 processes) in 2004-05 and all 32 of the Part B monitoring 
processes in 2006-07 included a Corrective Action and Enforcement component. For those 
processes that included the component, the average estimated percentage of the Corrective 
Action and Enforcement elements present was 72.5 in 2004-05 and 70.7 in 2006-07. Twenty- 
three of the 34 Part B monitoring processes in 2004-05 and 2 1 of the 32 of the Part B monitoring 
processes in 2006-07 included an Improvement Planning and Implementation component. For 
those processes that included the component, the average estimated percentage of the 
Improvement Planning and Implementation elements present was 55.4 in 2004-05 and 61.5 in 
2006-07. Fewer than one-third of the Part B monitoring processes in 2004-05 (11 of the 34 
processes) and in 2006-07 (9 of the 32 processes) included a Reassessment component. For those 
processes that included the component, the average estimated percentage of the Reassessment 
elements present was 52.0 in 2004-05 and 41.6 in 2006-07. 

Table 34. Part B average component scores and standard deviations for framework 
components: 2004-05 and 2006-07 



Component 


N 


2004-05 

Mean 


SD 


N 


2006-07 

Mean 


SD 


Problem Identification 


34 


58.6 


28.1 


32 


71.4 


19.7 


Problem Investigation 


16 


72.1 


24.3 


17 


71.9 


22.2 


Corrective Action and Enforcement 


32 


72.5 


16.4 


32 


70.7 


22.4 


Improvement Planning and Implementation 


23 


55.4 


29.7 


21 


61.5 


21.0 


Reassessment 


11 


52.0 


33.9 


9 


41.6 


25.9 



NOTE: There were a total of 34 Part B monitoring processes in 2004-05 and 32 Part B monitoring processes in 
2006-07. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

With regard to the standard deviations of the average component scores, if the average 
component scores were normally distributed, 50 then one would expect standard deviations in the 
range of 10 to 12 points. However, the standard deviations around the average component scores 
for Part B ranged from 16.4 to 33.9 in 2004-05 and from 19.7 to 25.9 in 2006-07 (see table 34). 
These high standard deviations reflect a flat distribution, which means that there was wide 
variability in the estimated percentage of elements present for each component. 



50 A normal distribution is a pattern of distribution of data that resembles a bell-shaped curve. 
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Part C Average Component Scores 

There were a total of 24 Part C monitoring processes in 2004-05 and 28 Part C monitoring 
processes in 2006-07. All of the Part C monitoring processes in 2004-05 and 2006-07 included a 
Problem Identification component (see table 35). The average estimated percentage of the 
Problem Identification elements present was 59.0 in 2004-05 and 68.8 in 2006-07. One-third or 
fewer of the Part C monitoring processes in 2004-05 (8 of the 24 processes) and 2006-07 (8 of 
the 28 processes) included a Problem Investigation component. For those processes that included 
the component, the average estimated percentage of the Problem Investigation elements present 
was 60.1 in 2004-05 and 63.4 in 2006-07. Most of the Part C monitoring processes in 2004-05 
(21 of the 24 processes) and 2006-07 (23 of the 28 processes) included a Corrective Action and 
Enforcement component. For those processes that included the component, the average estimated 
percentage of the Corrective Action and Enforcement elements present was 75.3 in 2004-05 and 
73.8 in 2006-07. Fewer than half of the Part C monitoring processes in 2004-05 (11 of the 24 
processes) and 2006-07 (12 of the 28 processes) included an Improvement Planning and 
Implementation component. For those processes that included the component, the average 
estimated percentage of the Improvement Planning and Implementation elements present was 
57.3 in 2004-05 and 54.1 in 2006-07. Twelve of the 24 Part C monitoring processes in 2004-05 
and 10 of the 28 Part C monitoring processes in 2006-07 included a Reassessment component. 
For those processes that included the component, the average estimated percentage of the 
Reassessment elements present was 42.9 in 2004-05 and 34.3 in 2006-07. 

Table 35, Part C average component scores and standard deviations for framework 
components: 2004-05 and 2006-07 



Component 


N 


2004-05 

Mean 


SD 


N 


2006-07 

Mean 


SD 


Problem Identification 


24 


59.0 


21.0 


28 


68.8 


21.2 


Problem Investigation 


8 


60.1 


29.7 


8 


63.4 


25.5 


Corrective Action and Enforcement 


21 


75.3 


10.4 


23 


73.8 


20.6 


Improvement Planning and Implementation 


11 


57.3 


19.0 


12 


54.1 


19.7 


Reassessment 


12 


42.9 


14.7 


10 


34.3 


29.9 



NOTE: There were a total of 24 Part C monitoring processes in 2004-05 and 28 Part C monitoring processes in 
2006-07. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 



Standard deviations for the average Part C component scores ranged from 10.4 to 29.7 in 2006 
and from 19.7 to 29.9 in 2006-07 (see table 35). As with the average Part B component scores, 
these large standard deviations for the Part C component scores indicate wide variability in the 
estimated percentage of elements present for each component. 
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Individual Elements and Part B and Part C Monitoring Processes 

The previous section reported on component scores for the framework, by looking at the average 
estimated percentage of elements that were present for each component, averaged across the Part 
C and Part C monitoring processes. This section examines the individual elements that made up 
each component and the number and proportion of state Part B and Part C monitoring processes 
in which each individual element was present. 

Below each component is examined in turn, with a focus on the extremes — those individual 
elements that were present in two-thirds or more (>67%) of the processes and those that were 
present in less than one-third (<33%) of the processes. These percentages are admittedly 
arbitrary, but were chosen to highlight the elements from the framework that were present most 
and least often. 

Analyses are based on the 65 elements that were used in both rounds of site visits. As explained 
in chapter 4, the number of elements was reduced from 137 in the first round to 65 in the second 
round to provide a more manageable and efficient set of ratings, and reduce the burden of data 
collection on the states. For these analyses, an element was counted as present if it was rated 
either as a “Yes” (on elements scored as Yes/No) or a “2” (on elements scored 0/1/2, where “0” 
meant not present, “1” meant sometimes present, and “2” meant always present), and it was 
counted as absent if it was rated either a “No” (on elements scored Yes/No) or a “0” or “1” (on 
elements scored 0/1/2). 51 

Problem Identification 



As discussed earlier in this chapter, all Part B and Part C monitoring processes were observed to 
have the Problem Identification component in both 2004-05 and 2006-07 (see tables 8 and 21). 
Problem identification has three subcomponents — Indicator and Target Setting, Indicator Data 
Collection and Analysis, and Problem Detection. The number of elements associated with each 
subcomponent varies. Overall, 23 elements were associated with this component for Part B and 
22 elements for Part C (see table 36) in both rounds of site visits. The subcomponents include 
several elements related to stakeholders, including the involvement of various types of 
stakeholders in selecting indicators and targets and the dissemination of findings to stakeholders 
(A1A1-A1C, and C4A-C4C). Other elements address technical features of the monitoring 
process, such as the presence of realistic and rigorous targets for the indicators, well-documented 
and credible data collection methodologies, procedures to ensure collection of valid and reliable 
data, and reporting on findings related to LEA/EIS performance (A3A-C3A). 



51 See chapter 4 for more details on the rating of elements. Of the 65 elements used in the analyses, 55 were rated as 
“Yes/No” and 10 were rated as “0/1/2.” 

52 One element, A1B, pertaining to involvement of state agency leadership figures from education, was not 
applicable to Part C monitoring processes. 
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Table 36. Number of state monitoring processes in which the Problem Identification 
elements were present, by part: 2004-05 and 2006-07 



Element 


Part B 

2004-05 2006-07 

(N= 34) (N= 32) 


Part C 

2004-05 2006-07 

(N= 24) (N=2S) 


Indicator and Target Setting: 

A1A1. Stakeholder committee includes parents of children with 
disabilities or representatives from advocacy groups. 


22 


21 


13 


19 


A1A2. Parents of children with disabilities or the representatives 
from advocacy groups represent the diversity that exists in the 
state. 


18 


19 


9 


14 


A1A3. Materials for stakeholders are tailored to improve 

understanding for those who need it to permit their informed 
participation in the process. 


14 


17 


8 


14 


A1A5. Stakeholder committee includes direct service personnel 
(i.e., teachers, service providers, and principals). 


20 


22 


14 


20 


A1B. State agency leadership figures in general education (i.e., state 
superintendent, state school board members) are informed 
(aware) of selected indicators and targets. 


28 


29 


t 


t 


A1C. Leadership figures in key governmental positions (i.e., 

representative from the governor’s office, representative from 
the education or health committee of the state legislature) are 
informed (aware) of selected indicators and targets. 


18 


13 


12 


12 


A3A. Indicator definitions are clear and unambiguous. 


22 


29 


17 


23 


A3B. Indicators are measurable. 


27 


29 


17 


24 


A4A. Target setting is accomplished through a systematic process. 


13 


17 


3 


15 


A4B. Targets are documented. 


15 


19 


3 


15 


Indicator Data Collection and Analysis: 

B3A2. Written documentation exists describing the data collection 
methodologies, including site visit plans (if indicator data 
collection involves site visits to LEAs/EIS programs). 


27 


23 


20 


18 


B4A. Procedures for coding and scoring data are documented. 


21 


25 


15 


13 


B4C. Procedures are in place to monitor the quality of the data 
collected. 


18 


22 


12 


20 


B4D. Data provided directly to the state by LEAs/EIS programs are 
verified for accuracy. 


18 


25 


10 


13 


B5B. Analyses are conducted to assess performance on the 
indicators/targets. 


24 


30 


22 


21 


Problem Detection: 

CIA. Findings reflect performance in relation to specific targets. 


17 


29 


18 


22 


C1C. Findings are adequately supported by data. 


20 


28 


19 


26 


C2A. State reports focus on the specified indicators and targets (i.e., 
discusses performance in relation to targets). 


14 


20 


11 


15 


C2C. State reports describe the methods and procedures used to 
identify problems. 


12 


14 


6 


11 
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Table 36. Number of state monitoring processes in which the Problem Identification 
elements were present, by part: 2004-05 and 2006-07 (continued) 





Part B 


Part C 


Element 


2004-05 

(7V=34) 


2006-07 
(AH 32) 


2004-05 
(AH 24) 


2006-07 

(AH28) 


C3A. Local-level reports focus on the specified indicators and 
targets (i.e., discusses performance in relation to targets). 


24 


20 


20 


21 


C4A. State-level problem identification reports are disseminated to 
stakeholders. 


11 


16 


9 


17 


C4B. Local level problem identification reports are disseminated to 
local stakeholders beyond the LEA/EIS program 
administrators. 


9 


11 


5 


10 


C4C. Opportunities are provided for stakeholders to discuss state- 
level problem identification reports with SEA/lead agency 
staff. 


11 


17 


10 


15 



f Not applicable. 



NOTE: The Ns in parentheses represent the numbers of Part B and Part C monitoring processes that had a Problem 
Identification component in 2004-05 and 2006-07. For Part B, there were a total of 34 processes in 2004-05 and 32 
processes in 2006-07. For Part C, there were a total of 24 processes in 2004-05 and 28 processes in 2006-07. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Problem Identification: Part B Monitoring Processes 

In 2004-05, a total of 5 of the 23 elements (22%) were each present in at least two-thirds of the 
34 Part B monitoring processes. One element involved stakeholders. Involvement of state agency 
leadership figures from general education in indie ator/target setting (A IB) was present in 82 
percent of Part B monitoring processes. Other elements that were present in 67 percent or more 
of the Part B monitoring processes all pertained to technical features of the monitoring process. 
They were measurable indicators (A3B, present in 79%), written documentation of data 
collection methodologies (B3A2, 79%), analyses conducted to assess perfonnance on 
indicators/targets (B5B, 71%), and focus of local reports on indicators/targets (C3A, 71%). 

Three of the 23 elements (13%) were each present in less than one -third of the 34 Part B 
monitoring processes in 2004-05. All three were related to stakeholders. These three elements 
were dissemination of local-level problem identification reports to stakeholders (C4B, 26%), 
dissemination of state-level problem identification reports to stakeholders (C4A, 32%), and 
opportunities for stakeholders to discuss state-level reports (C4C, 32%). 

In 2006-07, a total of 1 1 of the 23 elements (48%) were each present in at least two-thirds of the 
32 Part B monitoring processes. The element for analyses conducted to assess performance on 
indicators/targets (B5B) was present in 30 of the 32 Part B monitoring processes (94%). Three 
other elements from the earlier time period also were frequently present (A1B, 91%; A3B, 91%; 
and B3A2, 72%), plus seven additional ones. These seven were inclusion of direct service 
personnel as stakeholders (A1A5, 69%), clear and unambiguous indicators (A3 A, 91%), coding 
and scoring procedures for data (B4A, 78%), procedures in place to monitor data quality (B4C, 
69%), data verification (B4D, 78%), findings reflecting performance (CIA, 91%), and data 
support for findings (C1C, 88%). 
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In 2006-07, no Problem Identification elements were present in fewer than one-third of the 32 
Part B monitoring processes. 

Problem Identification: Part C Monitoring Processes 

In 2004-05, a total of 7 of the 22 Problem Identification elements (32%) were each present in at 
least two-thirds of the 24 Part monitoring C processes. All dealt with technical features of the 
monitoring process. These seven elements included conducting data analyses on indicators/ 
targets (B5B, 92%), clear and unambiguous indicators (A3 A, 71%), measurable indicators (A3B, 
71%), written documentation of data collection methodologies (B3A2, 83%), findings reflecting 
performance on targets (CIA, 75%), data support for findings (C1C, 79%), and focus of local- 
level reports on indicators and targets (C3A, 83%). 

Four of the 22 elements (18%) were each present in fewer than one-third of the 24 Part C 
monitoring processes in 2004-05. The elements present in fewer than one-third of Part C 
monitoring processes were systematic target setting (A4A, 13%), documentation of targets (A4B, 
13%), the description of methods and procedures in state-level problem identification reports 
(C2C, 25%), and dissemination of local-level problem identification reports to stakeholders 
(C4B, 21%). 

In 2006-07, 9 of the 22 elements (41%) were each present in at least two-thirds of the 28 Part C 
monitoring processes. Elements that were present in at least two-thirds of the Part C monitoring 
processes in 2006-07 included six of those cited for 2004-05 (C1C, 93%; A3 A, 82%; A3B, 86%; 
B5B, 75%; CIA, 79%; and C3A, 75%) along with three additions. The additional elements were 
inclusion of parents or advocates as stakeholders (A1A1, 68%), inclusion of direct service 
personnel as stakeholders (A1A5, 71%), and procedures to monitor data quality (B4C, 71%). 

In 2006-07 no Problem Identification elements were present in fewer than one-third of the 28 
Part C monitoring processes. 

Problem Investigation 

Overall, 1 1 elements for the Problem Investigation component were used in both rounds of site 
visits (see table 37). They included elements that address input and participation from key 
stakeholder groups (D1A1-D1C) and elements that address technical features of problem 
investigation, including credible problem investigation teams, well-documented data collection 
methodologies, procedures to ensure collection of valid and reliable data, use of appropriate data 
analysis techniques, and defensible findings (D2B-D8A). 

As discussed earlier in this chapter, not all Part B and Part C monitoring processes included a 
Problem Investigation component in either time period (see tables 8 and 21). In 2004-05, a total 
of 16 of 34 Part B monitoring processes (47%) and 8 of 24 Part C monitoring processes (33%) 
included this component. In 2006-07, some 17 of 32 Part B monitoring processes (53%) and 8 of 
28 Part C monitoring processes (29%) included this component. This section, therefore, 
examines just those processes that included the Problem Investigation component, focusing on 
which elements were often or infrequently present (i.e., present in at least two-thirds or more or 
present in less than one-third). 
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Table 37. Number of state monitoring processes in which the Problem Investigation 
elements were present, by part: 2004-05 and 2006-07 



Element 


Part B 

2004-05 2006-07 

(«=16) (n =17) 


Part C 

2004-05 2006-07 

(n =8) ( n =8) 


D1A1. Input is sought from parents of children with disabilities 
or representatives from advocacy groups. 


12 


12 


2 


4 


D1A3. Materials used to seek input from stakeholders are 
tailored to improve understanding and facilitate their 
contribution to the problem investigation process. 


9 


11 


2 


3 


D1C. A state leadership figure from special education or early 
intervention (e.g., state director of special education. Part C 
lead agency coordinator, the state monitoring director) are 
informed (are aware) of the problem investigation process. 


14 


16 


7 


7 


D2B. Individuals conducting problem investigations have 
appropriate training. 


15 


16 


7 


8 


D3B. Procedures are in place to minimize interruptions/ 

disruptions to school, program, or district routines when 
collecting data. 


14 


15 


3 


4 


D4B 1 . Written documentation exists describing the purpose for 
collecting the data. 


15 


16 


5 


7 


D5A. Procedures for coding and scoring data are documented. 


12 


10 


5 


4 


D5B. Procedures are in place to monitor the quality of the data 
collected. 


9 


12 


6 


6 


D5C. Data are verified for accuracy. 


8 


11 


6 


5 


D6B. Data variability is examined. 


8 


5 


3 


2 


D8A. Findings focus directly on the identified problems under 
investigation. 


13 


14 


6 


6 



NOTE: The ns in parentheses represent the numbers of Part B and Part C monitoring processes that had a Problem 
Investigation component in 2004-05 and 2006-07. For Part B, there were a total of 34 processes in 2004-05 and 32 
processes in 2006-07. For Part C, there were a total of 24 processes in 2004-05 and 28 processes in 2006-07. 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Problem Investigation: Part B Monitoring Processes 

In 2004-05, a total of 16 of the 34 Part B monitoring processes had a Problem Investigation 
component (see table 8). Seven of the 1 1 elements (64%) were each present in at least two-thirds 
of the Part B monitoring processes observed to have the Problem Investigation component. In 
this time period, two elements — training for individuals conducting problem investigation (D2B) 
and documentation of data collection purposes (D4B1) — were present in 15 of the 16 Part B 
monitoring processes with a Problem Investigation component (94%). The other elements 
present in at least two-thirds of the Part B monitoring processes with this component were input 
from parents or advocates (D1A1, 75%), infonning state leadership figures (D1C, 88%), 
procedures for minimizing disruption during data collection (D3B, 88%), coding and scoring 
procedures (D5A, 75%), and focusing findings on identified problems (D8A, 81%). 
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None of the Problem Investigation elements were present in fewer than one-third of the 16 Part B 
monitoring processes with Problem Investigation in 2004-05. 

In 2006-07, 17 of the 32 Part B monitoring processes had a Problem Investigation component 
(see table 8). A total of 7 of the 1 1 elements (64%) were each present in at least two-thirds of the 
Part B monitoring processes with this component. In this time period, three elements cited in the 
earlier time period were present in 16 of the 17 Part B monitoring processes (94%) that had 
Problem Investigation — informing state leadership figures (D1C), training for individuals 
conducting problem investigation (D2B), and documentation of data collection purposes, 
(D4B1). Three other elements also repeated from 2004-05 (D1A1, 71%; D3B, 88%; and D8A, 
82%). One additional element, procedures for monitoring data quality (D5B, 71%), also was 
present in at least two-thirds of Part B monitoring processes with this component in 2006-07. 

In 2006-07, only 1 of the 1 1 elements (9%) was present in fewer than one-third of the 17 Part B 
monitoring processes that included Problem Investigation. This element — examination of data 
variability (D6B) — was present in 5 out of 17 processes (29%). 

Problem Investigation: Part C Monitoring Processes 

In 2004-05, a total of 8 of the 24 Part C monitoring processes had a Problem Investigation 
component (see table 21). Five of the 1 1 elements (45%) were each present in at least two-thirds 
of the Part C monitoring processes that included Problem Investigation. These five elements 
dealt with informing state leadership figures of problem investigation (D1C, 88%), training for 
individuals conducting problem investigation (D2B, 88%) procedures to monitor data quality 
(D5B, 75%), verification of data accuracy (D5C, 75%), and focusing findings on identified 
problems (D8A, 75%). 

Two of 11 elements (18%) were present in fewer than one-third of the eight Part C monitoring 
processes with the Problem Investigation component. These two elements, input from parents 
(D1A1) and tailoring materials to facilitate stakeholder input (D1A3) were each present in two of 
eight processes (25%). 

In 2006-07, a total of 8 of the 28 Part C monitoring processes had a Problem Investigation 
component (see table 21). Five of the 1 1 elements (45%) were each present in at least two-thirds 
of the Part C monitoring processes with Problem Investigation. The element for training those 
conducting problem investigations (D2B) was present in all eight Part C monitoring processes 
(100%). This and three of the other elements (D1C, 88%; D5B, 75%; and D8A, 75%) were the 
same ones present in at least two-thirds of the Part C monitoring processes in 2004-05. One 
additional element was present in 67 percent of the processes with Problem Investigation in 
2006-07 — documentation of purposes for data collection (D4B1, 88%). 

In 2006-07, only 1 of 1 1 elements (9%) was present in fewer than one -third of the eight Part C 
monitoring processes with a Problem Investigation component. This element, examination of 
data variability (D6B), was present in two of the eight Part C processes (25%). 
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Corrective Action and Enforcement 



According to the framework for monitoring presented in chapter 3, states have two general ways 
of responding to the problems that they find through their monitoring procedures. One way, 
Corrective Action and Enforcement, is required in the law whenever noncompliance is identified. 
The other is Improvement Planning and Implementation, described below. Seven elements for 
the Corrective Action component were used in both rounds of site visits (see table 38). Two of 
the elements are stakeholder-related, addressing dissemination of reports to stakeholders and 
opportunities for local staff to discuss them with state officials (E2A and E2B), while the other 
five elements relate to the content of CAPs and the documentation of enforcement procedures 
(E1A1 - E1B2, E3A1, and E3A2). 

As discussed earlier in this chapter, not all Part B and Part C monitoring processes included the 
Corrective Action and Enforcement component in both time periods (see tables 8 and 21). In 
2004-05, 32 of the 34 Part B monitoring processes (94%) and 21 of the 24 Part C monitoring 
processes (88%) included Corrective Action and Enforcement; in 2006-07, all 32 Part B 
monitoring processes (100%) and 23 of the 28 Part C monitoring processes (82%) included this 
component. 

Table 38. Number of state monitoring processes in which the Corrective Action and 
Enforcement elements were present, by part: 2004-05 and 2006-07 





Part B 


Part C 


Element 


2004-05 

(n=32) 


2006-07 

(«=32) 


2004-05 

(77=21) 


2006-07 

(77=23) 


E1A1. CAPs specify all problem areas. 


28 


28 


21 


21 


E1B1. Steps/components of the CAPs are clearly delineated for 
each identified problem. 


26 


24 


18 


17 


E1B2. Timelines are included for each step/component of the 
CAPs. 


31 


22 


18 


18 


E2A. CAPs are disseminated to local stakeholders beyond the 
LEA/EIS program administrators. 


3 


4 


3 


5 


E2B. Opportunities are provided for LEA/EIS program personnel 
to discuss CAPs with SEA/lead agency staff. 


29 


28 


21 


22 


E3A1. The SEA/lead agency has written general enforcement 
procedures. 


24 


28 


15 


19 


E3A2. General enforcement procedures delineate specific actions 
that will be taken by the SEA/lead agency if a CAP is not 
implemented. 


21 


25 


12 


17 



NOTE: The ns in parentheses represent the numbers of Part B and Part C monitoring processes that had a Corrective 
Action and Enforcement component in 2004-05 and 2006-07. For Part B, there were a total of 34 processes in 2004- 
OS and 32 processes in 2006-07. For Part C, there were a total of 24 processes in 2004-05 and 28 processes in 2006-07. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Corrective Action and Enforcement: Part B Monitoring Processes 

In 2004-05, a total of 32 of the 34 Part B monitoring processes had a Corrective Action and 
Enforcement component (see table 8). Five of the seven elements (71%) were each present in at 
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least two-thirds of the Part B monitoring processes that included Corrective Action and 
Enforcement. The elements present in two-thirds or more of these Part B monitoring processes 
included three pertaining to the content of CAPs (E1A1, 88%; E1B1, 81%; and E1B2, 97%), 
providing opportunities for LEA personnel to discuss CAPs (E2B, 91%), and written general 
enforcement procedures (E3A1, 75%). 

One of the seven elements (14%) was not present in at least one-third of the 32 Part B 
monitoring processes with Corrective Action and Enforcement. This element, dissemination of 
CAPs to local stakeholders (E2A), was present in 3 of 32 processes (9%) in 2004-05. 

In 2006-07, all 32 of the Part B monitoring processes had a Corrective Action and Enforcement 
component (see table 8). Six out of seven elements (86%) were each present in at least two-thirds 
of the Part B monitoring processes with Corrective Action and Enforcement. During this time 
period, opportunities for LEA/EIS personnel to discuss CAPs (E2B), CAPs that specify all 
problem areas (E1A1,) and written general enforcement procedures (E3A1) were present in 28 of 
32 Part B monitoring processes (88%). The other elements present in at least two-thirds of Part B 
monitoring processes were the same as in 2004-05 (E1B1, 75%; and E1B2, 69%) with one 
addition — delineation of specific enforcement when CAPs are not implemented (E3A2, 78%). 

One of the seven elements (14%) for Corrective Action and Enforcement was present in less than 
one-third of Part B monitoring processes with this component. This element, dissemination of 
CAPs to stakeholders (E2A), was present in 4 of the 32 Part B monitoring processes (13%) in 
2006-07. 

Corrective Action and Enforcement: Part C Monitoring Processes 

In 2004-05, a total of 21 of the 24 Part C monitoring processes had a Corrective Action and 
Enforcement component (see table 21). Five of the seven elements (71%) were each present in at 
least two-thirds of Part C monitoring processes that included Corrective Action and 
Enforcement. Two elements — specification of all problem areas in CAPs (E1A1) and 
opportunities for LEA/EIS personnel to discuss CAPs (E2B) — were present in all 21 of these 
Part C monitoring processes (100%). The other elements present in two-thirds or more of these 
Part C monitoring processes dealt with the content of CAPs (E1B1, 86% and E1B2, 86%) and 
written general enforcement procedures (E3A1, 71%). 

One of the seven elements (14%) was not present in at least one-third of the 21 Part C 
monitoring processes with Corrective Action and Enforcement — dissemination of CAPs to 
stakeholders (E2A). It was present in 3 of the 21 Part C monitoring processes (14%) in 2004-05. 

In 2006-07, a total of 23 of the 28 Part C monitoring processes had a Corrective Action and 
Enforcement component (see table 21). Six of the seven elements (86%) were each present in at 
least two-thirds of the Part C monitoring processes that included Corrective Action and 
Enforcement. During this time period, the element addressing opportunities for LEA/EIS 
personnel to discuss CAPs (E2B) was present in 22 of 23 Part C monitoring processes (96%). 
The other elements were the same as in 2004-05 (E1A1, 91%; E1B1, 74%; E1B2, 78%; and 
E3A1, 83%), with one addition — specific enforcement when CAPs are not implemented (E3A2, 
74%). 
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The only element out of seven (14%) that was infrequently present was dissemination of CAPs 
to stakeholders (E2A). It was present in 5 of the 23 Part C monitoring processes with Corrective 
Action and Enforcement (22%) in 2006-07. 

Improvement Planning and Implementation 

Overall, 1 5 elements for the Improvement Planning and Implementation component were used in 
both rounds of site visits (see table 39). Several of these elements address involvement of and 
communication with key stakeholders, including LEAs/EIS programs (F1A1, F1A3, F4A-F4C, 
G1A1, and G2B). The others address technical and resource issues, including the contents of 
improvement plans, resources to implement improvement plans, and documentation to indicate 
improvement plans were implemented as intended (F2A1-F3C and G2A). 

Not all Part B and Part C processes included an Improvement Planning and Implementation 
component (see tables 8 and 21). In 2004-05, 23 of the 34 Part B monitoring processes (68%) 
and 1 1 of the 24 monitoring Part C monitoring processes (46%) included Improvement Planning 
and Implementation; in 2006-07, 21 of the 32 Part B monitoring processes (66%) and 12 of the 
28 Part C monitoring processes (43%) included this component. 

Table 39. Number of state monitoring processes in which the Improvement Planning and 
Implementation elements were present, by part: 2004-05 and 2006-07 



Element 


Part B 

2004-05 2006-07 

(77=23) (/7=21) 


PartC 

2004-05 2006-07 

(77=1 1) (77=12) 


Improvement Planning: 

F1A1. Input is sought from parents of children with disabilities or 
representatives from advocacy groups. 


10 


12 


5 


4 


F1A3. Materials used to seek input from stakeholders are tailored 
to improve understanding and facilitate their contribution to 
the improvement planning process. 


6 


9 


3 


4 


F2A1. Improvement plans specify all problem areas. 


18 


19 


11 


8 


F2A2. Written improvement plans address each problem area. 


19 


18 


11 


8 


F2B1. Steps/components of the improvement plans are clearly 
delineated for each identified problem. 


17 


16 


9 


8 


F3A2. Improvement plans include documentation of research and 
evidence -based practices on which the plans are based. 


7 


1 


1 


0 


F3B1. Plans specify TA to support implementation. 


18 


13 


8 


7 


F3B2. The plan specifies financial support for implementation. 


11 


6 


2 


1 


F3C. Plans describe the ways that implementation will be 
monitored by the SEA/lead agency. 


11 


13 


5 


8 


F4A. Improvement plans are disseminated to local stakeholders 
beyond the LEA/EIS program administrators. 


5 


9 


4 


5 


F4B. Opportunities are provided for LEA/EIS program personnel 
to discuss improvement plans with SEA/lead agency staff. 


22 


20 


10 


11 


F4C. Opportunities are provided for local stakeholders to discuss 
improvement plans with SEA/lead agency staff. 


8 


13 


5 


7 
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Table 39. Number of state monitoring processes in which the Improvement Planning and 
Implementation elements were present, by part: 2004-05 and 2006-07 
(continued) 



Element 


Part B 

2004-05 2006-07 

(«=23) (n= 21) 


Part C 

2004-05 2006-07 

(n=ll) (n=12) 


Improvement Plan Implementation: 

G1A1. Meetings are held with SEA/lead agency to discuss local 
implementation. 


15 


18 


6 


9 


G2A. Documentation confirms that plans were followed or 
changes in plans justified. 


17 


16 


9 


8 


G2B. SEA/lead agency provides follow-up and feedback to 
LEAs/EIS programs on a regular basis concerning their 
progress in implementing improvement plans. 


12 


17 


7 


9 



NOTE: The ns in parentheses represent the numbers of Part B and Part C monitoring processes that had an 
Improvement Planning and Implementation component in 2004-05 and 2006-07. For Part B, there were a total of 34 
processes in 2004-05 and 32 processes in 2006-07. For Part C, there were a total of 24 processes in 2004-05 and 28 
processes in 2006-07. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Improvement Planning and Implementation: Part B Monitoring Processes 

In 2004-05, a total of 23 of the 34 Part B monitoring processes had an Improvement Planning 
and Implementation component (see table 8). Six of the 15 elements (40%) for Improvement 
Planning and Implementation were each present in at least two-thirds of the Part B monitoring 
processes that included this component. In 2004-05, opportunities for LEA/EIS program 
personnel to discuss plans with the state (F4B) were present in 22 of these 23 Part B monitoring 
processes (96%). The other elements present in two-thirds or more of these Part B monitoring 
processes were specification of all problem areas in improvement plans (F2A1, 78%), written 
plans for all problems areas (F2A2, 83%), delineation of steps/components in the plans (F2B1, 
74%), specification of TA support (F3B1, 78%), and documentation to indicate plans were 
implemented (G2A, 74%). 

Three of the 15 elements (20%) were each present in fewer than one-third of the 23 Part B 
monitoring processes with Improvement Planning and Implementation in 2004-05. Elements 
present in fewer than one -third of the Part B monitoring processes in 2004-05 were 
dissemination of improvement plans to stakeholders (F4A, 22%), tailoring materials to facilitate 
stakeholder involvement (FI A3, 26%), and documentation of research and evidence-based 
practices in plans (F3A2, 30%). 

In 2006-07, 21 of the 32 Part B monitoring processes had an Improvement Planning and 
Implementation component (see table 8). Seven of the 15 elements (47%) were each present in at 
least two-thirds of the Part B monitoring processes with this component. Five elements were the 
same as in 2004-05 (F2A1, 90%; F2A2, 86%; F2B1, 76%; F4B, 95%; and G2A, 76%). Two 
additional elements, both dealing with state communication with FEAs, were present in 67 
percent of Part B monitoring processes with Improvement Planning and Implementation in 2006- 
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07; these were meetings to discuss local plan implementation (G1A1, 86%) and regular follow- 
up and feedback to LEAs (G2B, 81%). 

In 2006-07, a total of 2 of the 15 elements (13%) were each present in fewer than one -third of the 
21 Part B monitoring processes that included Improvement Planning and Implementation. One 
element, documentation of evidence-based practices (F3A2), was present in 1 of the 21 (5%) Part 
B monitoring processes. The other element present in fewer than 33 percent of the Part B 
monitoring processes in 2006-07 was specification of financial support (F3B2, 29%). 

Improvement Planning and Implementation: Part C Monitoring Processes 

In 2004-05, a total of 11 of the 24 Part C monitoring processes had an Improvement Planning 
and Implementation component (see table 21). Six of the 15 elements (40%) for Improvement 
Planning and Implementation were each present in at least two-thirds of the Part C monitoring 
processes with this component. Two elements — specification of all problem areas in 
improvement plans (F2A1) and written plans for all problems areas (F2A2) — were present in all 
11 Part C monitoring processes (100%). The other elements present in two-thirds or more of 
these Part C monitoring processes included delineation of steps/components in the plans (F2B1, 
82%), specification of TA support (F3B1, 73%), opportunities for FEA/EIS program personnel 
to discuss plans with the state (F4B, 91%), and documentation to indicate plans were 
implemented (G2A, 82%). 

Three of the 1 5 elements (20%) were each present in fewer than one-third of the 1 1 Part C 
monitoring processes with Improvement Planning and Implementation in 2004-05. The element 
for documentation of research and evidence-based practices in plans (F3A2) was present in 1 of 
1 1 Part C monitoring processes (9%). Other elements present in fewer than one-third of these 
Part C monitoring processes in 2004-05 dealt with tailoring materials to facilitate stakeholder 
involvement (FI A3, 27%) and specification of financial support (F3B2, 18%). 

In 2006-07, a total of 12 of the 28 Part C monitoring processes had an Improvement Planning 
and Implementation component (see table 21). Eight of the 15 elements (53%) were each present 
in at least two-thirds of the Part C monitoring processes that included Improvement Planning and 
Implementation. One element, opportunities for FEA/EIS program personnel to discuss plans 
with the state (F4B), was present in 11 of the 12 Part C monitoring processes (92%). This one 
and four other elements (F2A1, 67%; F2A2, 67%; F2B1, 67%; and G2A, 67%) were among 
those present in two-thirds of the Part C monitoring processes that included this component in 
2004-05. Of the three additional elements, two dealt with state communication with LEAs — 
meetings to discuss local plan implementation (G1A1, 75%) and regular follow-up and feedback 
to LEAs (G2B, 75%). The other was inclusion of state plans for monitoring implementation 
(F3C, 67%). 

In 2006-07, 2 of the 15 elements (13%) were each present in fewer than one-third of the 12 Part 
C monitoring processes that included Improvement Planning and Implementation. The element 
for documentation of evidence-based practices (F3 A2) was not present in any of the 2 1 Part C 
monitoring processes in 2006-07. The other element that was present in fewer than one-third of 
the Part C monitoring processes dealt with specification of financial support (8%) (F3B2). 
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Reassessment 



The Reassessment component of the framework involves states’ revisiting what has been 
identified as problematic, following efforts to make corrections or improvements. Overall, nine 
elements for the Reassessment component were used in both rounds of site visits (see table 40). 
Four of these elements are stakeholder related, addressing dissemination of reports to 
stakeholders and opportunities for LEA/EIS program personnel and other stakeholders to discuss 
the reports (H3A-H3D). The other five elements address the contents of the reports and 
independent review of the findings (H2A-H2G). 

Not all Part B and Part C monitoring processes included a Reassessment component (see tables 8 
and 21). In 2004-05, 10 of the 34 Part B monitoring processes (29%) and 12 of the 24 Part C 
monitoring processes (50%) included Reassessment; in 2006-07, 9 of the 32 Part B monitoring 
processes (28%) and 10 of the 28 Part C processes (36%) included this component. 

Table 40. Number of state monitoring processes in which the Reassessment elements 
were present, by part: 2004-05 and 2006-07 



Element 


Part B 

2004-05 2006-07 

(«=11) (77=9) 


Part C 

2004-05 2006-07 

(77=12) (77=10) 


H2A. Reports focus on specified indicators and targets. 


7 


7 


11 


6 


H2B. Reports describe the data used to reassess 

performance/compliance on specified indicators and 
targets. 


7 


6 


7 


6 


H2C. Reports describe the methods and procedures used to 
reassess performance or compliance. 


8 


2 


7 


3 


H2F. Reassessments of LEA/EIS programs include a 
discussion of strengths and areas needing further 
improvement. 


4 


4 


4 


4 


H2G. Someone who did not write the report or collect the data 
reviews the reports. 


9 


2 


7 


4 


H3A. Reassessment reports are disseminated to local 

stakeholders beyond the LEA/EIS program administrators. 


3 


1 


0 


1 


H3B. Opportunities are provided for LEA/EIS program 

personnel to discuss reassessment reports with SEA/lead 
agency staff. 


9 


8 


11 


6 


H3C. Opportunities are provided for local stakeholders to 
discuss reassessment reports with SEA/lead agency staff. 


3 


3 


0 


2 


H3D. Findings from reassessments are disseminated to state- 
level stakeholders. 


4 


2 


2 


2 



NOTE: The ns in parentheses represent the numbers of Part B and Part C monitoring processes that had a 
Reassessment component in 2004-05 and 2006-07. For Part B, there were a total of 34 processes in 2004-05 and 32 
processes in 2006-07. For Part C, there were a total of 24 processes in 2004-05 and 28 processes in 2006-07. 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 
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Reassessment: Part B Monitoring Processes 



In 2004-05, 1 1 of the 34 Part B monitoring processes had a Reassessment component (see table 
8). Three of the nine Reassessment elements (33%) were each present in at least two-thirds of the 
Part B monitoring processes that included this component. An independent review of findings 
(H2G) and opportunities for LEA/EIS program personnel to discuss reassessment reports with 
the state (H3B) were present in 9 of the 1 1 Part B monitoring processes (82%) with this 
component. The element for description of methods and procedures in the reports (H2C, 73%) 
was also present in two-thirds or more of Part B monitoring processes with this component. 

Two of the nine elements (22%) were each present in fewer than one-third of the 1 1 Part B 
monitoring processes that included Reassessment in 2004-05. These were both stakeholder- 
related elements, present in 3 of 1 1 Part B monitoring processes (27%). One addressed 
dissemination of reassessment reports to local stakeholders (H3A) and the other, opportunities 
for local stakeholders to discuss the reports with the state (H3C). 

In 2006-07, a total of 9 of the 32 Part B monitoring processes had a Reassessment component 
(see table 8). Three of the nine Reassessment elements (33%) were each present in at least two- 
thirds of the Part B monitoring processes with this component. One of the three elements was 
repeated from 2004-05, as opportunities for LEA/EIS program personnel to discuss reassessment 
reports with the state (H3B) were present in eight of the nine Part B monitoring processes (89%). 
The other two elements were focusing reports on indicators/targets (H2A, 78%) and description 
of data for reassessment in reports (H2B, 67%). 

In 2006-07, four of the nine elements (44%) were each present in fewer than one-third of the 
nine Part B monitoring processes that included Reassessment. One element, dissemination of 
reports to local stakeholders (H3A), was present in one of nine Part B monitoring processes 
(11%). The other elements in this group were inclusion of methods and procedures in 
reassessment reports (H2C, 22%), independent review of findings (H2G, 22%), and 
dissemination of findings to state-level stakeholders (H3D, 22%). 

Reassessment: Part C Monitoring Processes 

In 2004-05, 12 of the 24 Part C monitoring processes had a Reassessment component (see table 
21). Two of the nine elements (22%) were each present in at least two-thirds of the Part C 
monitoring processes that included Reassessment. The elements for focusing reports on 
indicators/targets (H2A) and opportunities for LEA/EIS program personnel to discuss 
reassessment reports with the state (H3B) were present in 11 of the 12 Part C monitoring 
processes (92%) with this component. 

Three of the nine elements (33%) — all stakeholder related — were each present in fewer than one- 
third of the 12 Part C monitoring processes in 2004-05. The elements addressing dissemination 
of reassessment reports to local stakeholders (H3A) and opportunities for local stakeholders to 
discuss the reports with the state (H3C) were not present in any of the 12 Part C monitoring 
processes with Reassessment. Dissemination of findings to state-level stakeholders (H3D, 17%) 
was the other element present in fewer than one-third of the Part C monitoring processes. 
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In 2006-07, a total of 10 of the 28 Part C monitoring processes had a Reassessment component 
(see table 21). In this time period, none of the nine elements were present in at least two-thirds of 
the Part C processes that included Reassessment. 

Four of the nine elements (44%) were each present in fewer than one-third of the 10 Part C 
monitoring processes that included Reassessment — including the three elements that fell into this 
group in 2004-05. The element for dissemination of reassessment reports to local stakeholders 
(H3A) was present in 1 of 10 Part C monitoring processes (10%). Opportunities for local 
stakeholders to discuss the reports with the state (H3C) and dissemination of findings to state- 
level stakeholders (H3D) were present in two Part C monitoring processes (20%); methods and 
procedures for reassessment were described in reassessment reports (H2C) for three Part C 
monitoring processes (30%). 

Summary for Analysis of Framework Components and Elements 

This section examined the ratings of state monitoring processes on the elements associated with 
the framework, using two different analytical approaches. The data for these analyses came from 
the ratings completed by site visitors for the 20 Part and Part C monitoring systems in 2004-05 
and 2006-07. Overall, 58 monitoring processes were rated in 2004-05, and 60 monitoring 
processes were rated in 2006-07. 

Average Component Scores 

The average component scores can be interpreted as the average estimated percentage of 
elements present for those processes that included that component. The average component 
scores for the Part B monitoring processes ranged from 52.0 (Reassessment) to 72.5 (Corrective 
Action and Enforcement) in 2004-05 and from 41.6 (Reassessment) to 71.9 (Problem 
Investigation) in 2006-07. The average component scores for the Part C monitoring processes 
ranged from 42.9 (Reassessment) to 75.3 (Corrective Action and Enforcement) in 2004-05 and 
from 34.3 (Reassessment) to 73.8 (Corrective Action and Enforcement) in 2006-07. 

Presence of Individual Elements 



The second analysis examined the individual elements that made up each component and the 
number and proportion of elements that were present in Part B and Part C monitoring processes 
for each component. Individual elements for every component included a mix of elements related 
to stakeholder involvement and communication and elements related to technical and procedural 
features of the monitoring process. Technical and procedural features include methodology, 
content of reports, and enforcement. The discussion focused on elements that fell at the 
extremes — those that were present in two-thirds (67%) or more of the Part B or Part C 
monitoring processes and those that were present in less than one-third (<33%) of the processes. 

The number and percentage of elements that were present in two-thirds or more of all Part B and 
Part C monitoring processes varied from component to component and across years. For Part B, 
in 2004-05, 26 of the 65 elements (40%) from all components were each present in two-thirds or 
more of the Part B monitoring processes that included the relevant components; proportions of 
elements present in two-thirds or more of the Part B monitoring processes with a component 
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ranged from 22 percent (5 of 23 elements) for the Problem Identification component to 71 
percent (5 of 7 elements) for the Corrective Action and Enforcement component. In 2006-07, 34 
of the 65 elements (52%) were each present in two-thirds or more of the Part B monitoring 
processes with the relevant component, ranging from 33 percent (3 of 9 elements) for the 
Reassessment component to 86 percent (6 of 7 elements) for the Corrective Action and 
Enforcement component. 

For Part C, in 2004-05, a total of 25 of 64 elements' (39%) from all components were each 
present in two-thirds or more of the Part C monitoring processes that included the relevant 
components; proportions of elements present in two-thirds or more of the Part C monitoring 
processes with a component ranged from 22 percent (2 of 9 elements) for the Reassessment 
component to 71 percent (5 of 7 elements) for the Corrective Action and Enforcement 
component. In 2006-07, a total of 28 of 64 elements (44%) were each present in two-thirds or 
more of the Part C monitoring processes with the relevant component, with proportions ranging 
from 0 percent (0 of 9 elements) for the Reassessment component to 86 percent (6 of 7 elements) 
for the Corrective Action and Enforcement component. 

The number and percentage of elements that were present in less than one-third of all Part B and 
Part C monitoring processes also varied across components and time periods. For Part B, in 
2004-05, a total of 9 of the 65 elements (14%) from all components were each present in less 
than one-third of the Part B monitoring processes that included the relevant components; 
proportions of elements present in less than one-third of Part B monitoring processes with a 
component ranged from 0 percent (0 of 1 1 elements) for the Problem Investigation component to 
22 percent (2 of 9 elements) for the Reassessment component. In 2006-07, a total of 8 of the 65 
elements (12%) were each present in less than one-third of the Part B monitoring processes that 
had the relevant component, with proportions ranging from 0 percent (0 of 23 elements) for the 
Problem Identification component to 44 percent (4 of 9 elements) for the Reassessment 
component. 

For Part C, in 2004-05, a total of 13 of the 64 elements (20%) from all components were each 
present in less than one -third of the Part C monitoring processes that included the relevant 
components; proportions of elements present in less than one-third of Part C monitoring 
processes with a component ranged from 14 percent (1 of 7 elements) for the Corrective Action 
and Enforcement component to 33 percent (3 of 9 elements) for the Reassessment component. In 
2006-07, a total of 8 of the 64 elements (13%) were each present in less than one-third of the Part 
C monitoring processes that had the relevant component, with proportions ranging from 0 
percent (0 of 23 elements) for the Problem Identification component to 44 percent (4 of 9 
elements) for the Reassessment component. 



53 One element was not applicable to Part C, so the total number is 64, not 65 as for Part B. 
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6. Report Summary 



The three objectives of this study were (1) to provide a description of the nature and scope of 
states’ monitoring systems, (2) to describe states’ monitoring systems at two points in time, and 
(3) to create a framework to describe the variation that exists in state monitoring systems. To 
address these objectives, the study included two data collection efforts: (1) a mail survey and (2) 
site visits. For the mail survey data collection, the project team collected descriptive information 
from all 50 states and the District of Columbia about their Part B and Part C monitoring systems. 
Findings from the mail survey data collection were presented in an earlier report (Garrison- 
Mogren et al. 2008). 

This report focuses on the site visit data collection. For this data collection, the project team first 
developed a framework for monitoring that consisted of five monitoring components: Problem 
Identification, Problem Investigation, Corrective Action and Enforcement, Improvement 
Planning and Implementation, and Reassessment. The project team then conducted site visits to a 
systematic random sample of 20 states. The 20 states were visited once in 2006 to find out more 
about states’ Part B and Part C monitoring systems in place in 2004-05 and then a second time in 
2008 to find out more about their monitoring systems in place in 2006-07. 

There was substantial variability in the nature and design of states’ Part B and Part C monitoring 
systems and processes in 2004-05 and 2006-07. This chapter presents a summary of key findings 
from the analyses of the site visit data, including findings regarding the context for state 
monitoring, states’ approaches to monitoring, how states’ monitoring systems and processes 
mapped onto the framework, and the elements associated with the framework components that 
were present or absent from states’ monitoring processes. 

As noted throughout this report, several issues should be kept in mind when interpreting the 
findings from this study. First, the 20-state sample used for the site visit data collection was 
designed to capture the variability in states’ Part B and Part C monitoring systems, but the intent 
was not to generalize the findings to the entire country. Second, although data were collected at 
two points in time in order to capture the variability that exists between and within states’ 
monitoring systems, the study did not assess change or trends over time. Such analyses were 
considered premature primarily because the framework developed to guide data collection and 
analysis had not been validated or evaluated to determine if its adoption resulted in an improved 
monitoring system or better outcomes for children and families. Third, the site visit interviews 
collected retrospective infonnation, as site visits in 2006 collected information about the 
monitoring systems in place in 2004-05, and the site visits in 2008 collected information about 
the monitoring systems in place in 2006-07. The possibility that some interviewees may have 
inaccurately recalled policies and procedures for the study’s two data collection periods cannot 
be ruled out. Last, it should be noted that the framework was developed as a basis for describing 
the nature and design of states’ Part B and Part C monitoring systems; it has not been validated 
or evaluated in regard to whether the adoption of this framework resulted in an improved state 
monitoring system. 
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The Context for State Monitoring 

The report first reviews some of the contextual factors that may affect states’ Part B and Part C 
monitoring systems and processes. With regard to resources for monitoring, state agency staff in 
a number of states mentioned staffing shortages or turnover as a concern (four Part B monitoring 
systems in both 2004-05 and 2006-07 and four Part C monitoring systems in 2004-05 and seven 
Part C monitoring systems in 2006-07). Some states reported using consultants or contractors to 
help carry out their monitoring responsibilities — including 7 Part B monitoring systems and 10 
Part C monitoring systems in 2004-05 and 12 Part B and Part C monitoring systems in 2006-07. 

A number of state monitoring systems were in transition in each time period; this was the case 
for 9 Part B and Part C monitoring systems in 2004-05 and 12 Part B monitoring systems and 7 
Part C monitoring systems in 2006-07. Individual circumstances varied, ranging from states that 
were developing new monitoring processes and procedures while phasing out old ones to states 
that were fully implementing new monitoring processes for all LEAs/EIS programs. Some states 
in both 2004-05 and 2006-07 reported that they had worked with NCSEAM to develop one or 
more of their monitoring processes; this included 4 Part B monitoring systems in 2004-05 and 10 
Part B monitoring systems in 2006-07 and 4 Part C monitoring systems in both 2004-05 and 
2006-07. 

In both 20047-05 and 2006-07, Part B and Part C officials cited many other contextual factors 
that affected their monitoring, but few types of issues were cited by more than two states. Issues 
that were cited by more than two states in either time period included issues related to lawsuits, 
consent decrees, or compliance agreements with OSEP; administrative issues related to 
organizational structure or program responsibility; and financial issues. 



State Approaches to Monitoring 

The report provides an overview of states’ approaches to monitoring, including whether states’ 
monitoring systems included one process or multiple processes, the types of monitoring 
processes that made up states’ monitoring systems, and how often monitoring processes were 
initiated. 

Number of Monitoring Processes 

States’ Part B and Part C monitoring systems could be composed of either a single monitoring 
process or multiple monitoring processes. In both 2004-05 and 2006-07, half of the 20 selected 
states’ Part B monitoring systems were composed of multiple processes, and half were composed 
of single monitoring processes. In both 2004-05 and 2006-07, states’ Part C monitoring systems 
were typically made up of a single monitoring process (17 Part C monitoring systems and 13 Part 
C monitoring systems, respectively). 

In 2004-05, a total of 58 monitoring processes were identified in the 20 Part B and Part C 
monitoring systems; 34 of these were Part B monitoring processes, and 24 were Part C 
monitoring processes. In 2006-07, a total of 60 monitoring processes were identified; 32 of these 
were Part B monitoring processes, and 28 were Part C monitoring processes. 
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Process Types 

In both 2004-05 and 2006-07, four types of monitoring processes were identified: 54 

(1) performance-based state assessment; 

(2) schedule -based local assessment; 

(3) schedule-based state assessment; and 

(4) schedule-based hybrid assessment. 

States used these processes singly or in combination to carry out their monitoring 
responsibilities. 

For Part B, all four types of processes were represented among the processes observed in each 
year. Half of all Part B monitoring systems used multiple processes, using either a combination 
of schedule-based processes or a combination of schedule-based and performance-based 
processes. No Part B multi-process monitoring systems were observed to rely exclusively on 
performance-based assessments in either time period. In the Part B single-process monitoring 
systems, all four types of assessment were observed. 

In 2004-05, the majority of Part C monitoring processes (18 of 24 Part C monitoring processes) 
fell into the category of schedule-based state assessment, and only two processes were 
performance-based state assessments. In 2006-07, schedule-based state assessment processes 
were still the majority, but there were small increases in all the other categories. In addition, 35 
percent of Part C monitoring systems were using multiple processes; 15 percent were multiple 
process systems in 2004-05. All of the Part C single-process monitoring systems used either a 
local or state schedule-based assessment in 2004-05; in 2006-07, the single-process systems 
included all four types of monitoring processes. Similar to Part B, those Part C monitoring 
systems that combined processes relied either on schedule-based processes only or combined 
them with performance-based monitoring. None of the Part C multi-process monitoring systems 
used only perfonnance-based monitoring. 

How Often Monitoring Processes Were Initiated 

Continuous monitoring — that is, monitoring that was initiated more than once a year — was 
observed only in Part C monitoring systems. In 2004-05, a total of 4 of the 24 Part C monitoring 
processes were continuous; 5 of the 28 Part C monitoring processes were continuous in 2006-07. 

The remainder of the Part C monitoring processes and all the Part B monitoring processes were 
initiated annually, cyclically, or as performance dictated. All of these frequencies of initiation 
(except continuous) were represented among the 10 Part B single-process monitoring systems in 
both time periods. All frequencies were represented among the 13 Part C single-process 
monitoring systems in 2006-07, but performance-dictated schedules were not observed in the 17 
Part C single-process systems in 2004-05. In both time periods, the Part B and Part C multi- 



54 Definitions of the four monitoring process types can be found in table 3. 
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process monitoring systems used the different frequencies in various combinations; however, 
none of the Part B or Part C multi-process systems in either year used performance-dictated 
schedules only. 



Mapping Onto the Framework Components 

To describe the nature and design of states’ Part B and Part C monitoring systems, the project 
team developed a framework that consisted of five key monitoring components: Problem 
Identification, Problem Investigation, Corrective Action and Enforcement, Improvement 
Planning and Implementation, and Reassessment. This report describes the number of Part B and 
Part C monitoring systems and processes that had each of these components and then discusses 
the various ways in which states’ Part B and Part C monitoring processes approached that 
component. 

In both 2004-05 and 2006-07, few Part B monitoring processes (4 of the 34 Part B monitoring 
processes in 2004-05 and 4 of the 32 Part B monitoring processes in 2006-07) or Part C 
monitoring processes (3 of the 24 Part C monitoring processes in 2004-05 and 1 of the 28 Part C 
monitoring processes in 2006-07) within the 20 states included all five of the framework 
components. For Part B, the most common number of components in the states’ monitoring 
processes was three in 2004-05 (14 Part B monitoring processes) and four in 2006-07 (14 Part B 
monitoring processes). For Part C, in 2004-05, equal numbers of monitoring processes included 
either two components (7 of the 24 Part C monitoring processes) or four components (7 of the 24 
Part C monitoring processes); in 2006-07, the most common number of components that states’ 
monitoring processes included was three (10 of the 24 Part C monitoring processes). 

Problem Identification 

All of the Part B and Part C monitoring processes had procedures in place for identifying 
problems in both 2004-05 and 2006-07. With regard to the selection of indicators and targets to 
monitor local compliance or performance, in 2004-05, state agency staff reported that they 
identified their indicators and targets based on reviews of state and federal regulations or 
requirements, either with or without input from their state-level stakeholder group for all of the 
Part B and Part C monitoring processes (34 Part B and 24 Part C monitoring processes). In 2006- 
07, for many of the Part B and Part C monitoring processes (16 of the 32 Part B monitoring 
processes and 19 of the 28 Part C monitoring systems), state agency staff reported that they 
elected to use the OSEP SPP/APR indicators and targets instead of selecting and defining their 
own indicators as they had done in the past. Most noted (9 of the 16 Part B monitoring processes 
and 14 of the 19 Part C monitoring processes), though, that they also had reviewed their state 
regulations and added one or more indicators to ensure that they were meeting state-level 
requirements in addition to the federal requirements. 

Not all states selected both compliance and outcome indicators. In 2004-05, a total of 20 of the 
34 Part B monitoring processes and 6 of the 24 Part C monitoring processes included both 
compliance and outcome indicators; 9 of the 34 Part B monitoring processes and 18 of the 24 
Part C monitoring processes included compliance indicators only. In 2006-07, a total of 20 of the 
32 Part B monitoring processes and 15 of the 28 Part C monitoring processes included both 
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compliance and outcome indicators; 8 of the 32 Part B monitoring processes and 12 of the 28 
Part C monitoring processes included compliance indicators only. 

States’ Part B and Part C monitoring processes used data from different sources to assess 
performance on indicators. The most common sources of data for Part B were existing state or 
local data, such as data profdes or data for section 618 of IDEA (e.g., suspension/expulsion data, 
racial/ethnic data to assess disproportionality) to assess local performance (used by 25 of the 34 
Part B monitoring processes in 2004-05 and 26 of the 32 Part B monitoring processes in 2006- 
07) and record reviews (used in 23 of the 34 Part B monitoring processes in 2004-05 and 23 of 
the 32 Part B monitoring processes in 2006-07). The most common source of data for Part C was 
record review (used in 20 of the 24 Part C monitoring processes in 2004-05 and 22 of the 28 Part 
C monitoring processes in 2006-07). 

Problem Investigation 

According to the framework, the Problem Identification component indicates to the state that 
there is a problem with local compliance or performance, but it does not really speak to why the 
problem exists. In order to understand why the identified problems exist, states need to further 
investigate. Sixteen of the 34 Part B monitoring processes in 2004-05 and 17 of the 32 Part B 
monitoring processes in 2006-07 included a Problem Investigation component. Eight of the 24 
Part C monitoring processes in 2004-05 and 8 of the 28 Part C monitoring processes in 2006-07 
included a Problem Investigation component. The remaining Part B and Part C monitoring 
processes did not include a Problem Investigation component and, therefore, moved directly 
from identifying problems (i.e., Problem Identification) to addressing problems (i.e., Corrective 
Action and Enforcement and/or Improvement Planning and Implementation). 

For both Part B and Part C monitoring processes, problem investigations were typically carried 
out solely by state agency staff (1 1 of the 16 Part B monitoring processes in 2004-05 and 9 of the 
17 in 2006-07; 6 of the 8 Part C monitoring processes in 2004-05 and 5 of the 8 in 2006-07). In 
the remaining Part B and Part C monitoring processes, LEA/EIS program staff participated in 
problem investigations, either alone or in conjunction with state agency staff (5 of the 16 Part B 
monitoring processes in 2004-05 and 8 of the 17 in 2006-07; 2 of the 8 Part C monitoring 
processes in 2004-05 and 3 of the 8 in 2006-07). 

In both 2004-05 and 2006-07, in the majority of the states’ Part B and Part C monitoring 
processes that had a Problem Investigation component, the same approach was used to 
investigate problems across LEAs/EIS programs, regardless of the nature of the problems 
identified (10 of the 16 Part B monitoring processes in 2004-05 and 10 of the 17 Part B 
monitoring processes in 2006-07; 7 of the 8 Part C monitoring processes in 2004-05 and 5 of the 
8 Part C monitoring processes in 2006-07). In the remaining Part B and Part C monitoring 
processes, the exact approach or approaches used were based on the nature of the problem under 
investigation (6 of the 16 Part B monitoring processes in 2004-05 and 7 of the 17 Part B 
monitoring processes 2006-07; 1 of the 8 Part C monitoring processes in 2004-05 and 3 of the 8 
Part C monitoring processes in 2006-07). 
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Corrective Action and Enforcement 



States’ Part B and Part C monitoring processes typically addressed identified problems related to 
noncompliance through Corrective Action and Enforcement. Overall, most of the Part B and 
Part C monitoring processes in both years included this component (32 of the 34 Part B 
monitoring processes in 2004-05 and all 32 of the Part B monitoring processes in 2006-07; 21 of 
the 24 Part C monitoring processes in 2004-05 and 23 of the 28 Part C monitoring processes in 
2006-07). 

In a small number of Part B and Part C monitoring processes, state agency staff reported that 
they did not follow up with LEAs/EIS programs to ensure implementation of CAPs (8 of the 32 
Part B monitoring processes in 2004-05 and 4 of the 32 Part B monitoring processes in 2006-07; 
3 of the 2 1 Part C monitoring processes in 2004-05 and 1 of the 23 Part C monitoring processes 
in 2006-07.) For those Part B and Part C monitoring processes in which state agency staff 
followed up on CAPs (23 of the 32 Part B monitoring processes in 2004-05 and 28 of the 32 Part 
B monitoring processes in 2006-07; 18 of the 21 Part C monitoring processes in 2004-05 and 22 
of the 23 Part C monitoring processes in 2006-07), one of the more common follow-up 
approaches involved having the state monitoring team conduct on-site visits or reviews of data to 
determine whether corrections had been made (9 of the 23 Part B monitoring processes in 2004- 
OS and 17 of the 28 Part B monitoring processes in 2006-07; 11 of the 18 Part C monitoring 
processes in 2004-05 and 14 of the 22 Part C monitoring processes in 2006-07). 

Improvement Planning and Implementation 

Smaller numbers of states’ Part B and Part C monitoring processes addressed identified problems 
via Improvement Planning and Implementation. For Part B, 23 of the 34 Part B monitoring 
processes in 2004-05 and 21 of the 32 Part B monitoring processes in 2006-07 included 
Improvement Planning and Implementation. For Part C, 1 1 of the 24 Part C monitoring processes 
included this component in 2004-05, and 12 of the 28 Part C monitoring processes included it in 
2006-07. 

As with CAPs, in a small number of Part B and Part C monitoring processes, state agency staff 
reported that they did not follow up with LEAs/EIS programs to ensure implementation of 
improvement plans (8 of the 23 Part B monitoring processes in 2004-05 and 3 of the 21 Part B 
monitoring processes in 2006-07; 3 of 1 1 Part C monitoring processes in 2004-05 and 2 of the 12 
Part C monitoring processes in 2006-07). For those Part B and Part C monitoring processes in 
which state agency staff followed up on improvement plans (15 of the 23 Part B monitoring 
processes in 2004-05 and 17 of the 21 Part B monitoring processes in 2006-07; 8 of the 1 1 Part C 
monitoring processes in 2004-05 and 10 of the 12 Part C monitoring processes in 2006-07), one 
of the more common follow-up approaches involved having LEAs/EIS programs submit 
progress reports about improvement plan implementation (5 of the 23 Part B monitoring 
processes in 2004-05 and 14 of the 21 Part B monitoring processes in 2004-05; 6 of the 1 1 Part C 
monitoring processes in 2004-05 and 7 of the 12 Part C monitoring processes in 2006-07.) 
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Reassessment 



Half or fewer of the Part B monitoring processes ( 1 1 of the 34 Part B monitoring processes in 
2004-05 and 9 of the 32 Part B monitoring processes in 2006-07) and Part C monitoring 
processes (12 of the 24 Part C monitoring processes in 2004-05 and 10 of the 28 Part C 
monitoring processes in 2006-07) included procedures to reassess identified problems. That is, 
they often did not have procedures in place to assess whether the corrective actions and 
improvement strategies that were implemented actually ameliorated the identified problems. 
According to the framework, this component informs states whether the actions they have 
implemented to address the identified problems are working. Otherwise, states will not know if 
there is a need to either further investigate the problem or revise the strategies the LEAs/EIS 
programs have implemented. 



Analysis of the Framework Components and Elements 

Each of the framework components included key characteristics and elements that defined the 
component and identified aspects thought to be important to the component being carried out 
(see appendix B). During the first and second rounds of site visits, the site visit teams rated the 
elements associated with the framework components to detennine whether they were present or 
absent in states’ Part B and Part C monitoring processes in 2004-05 and 2006-07. This report 
used two different approaches to analyze these data. Findings for each of the analytical 
approaches are summarized below. 

Average Component Scores for Part B and Part C Monitoring Processes 

The first analysis presented the average estimated percentage of elements present for each 
component of the framework for states’ Part B and Part C monitoring processes in 2004-05 and 
2006-07. Only those monitoring processes that had the component were included in the 
calculation of the average component scores. For example, 1 1 Part B monitoring processes in 
2004-05 had a Reassessment component. For those 1 1 Part B monitoring processes, on average, 
52 percent of the elements associated with this component were present. 

In 2004-05, there were a total of 34 Part B monitoring processes; of these, all 34 included a 
Problem Identification component, 16 included a Problem Investigation component, 32 included 
a Corrective Action and Enforcement component, 23 included an Improvement Planning and 
Implementation component, and 1 1 included a Reassessment component. In 2006-07, there were 
a total of 32 Part B monitoring processes; of these, all 32 included a Problem Identification 
component, 17 included a Problem Investigation component, all 32 included a Corrective Action 
and Enforcement component, 21 included an Improvement Planning and Implementation 
component, and 9 included a Reassessment component. The average component scores ranged 
from 52.0 (Reassessment) to 72.5 (Corrective Action and Enforcement) in 2004-05 and from 
41.6 (Reassessment) to 71.9 (Problem Investigation) in 2006-07. More specifically, the average 
estimated percentage of elements present for those Part B monitoring processes that had each 
component was as follows: 

• 58.6 in 2004-05 and 71.4 in 2006-07 for the Problem Identification component; 
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• 72.1 in 2004-05 and 71.9 in 2006-07 for the Problem Investigation component; 

• 72.5 in 2004-05 and 70.7 in 2006-07 for the Corrective Action and Enforcement 
component;, 

• 55.4 in 2004-05 and 61.5 in 2006-07 for the Improvement Planning and 
Implementation component; and 

• 52.0 in 2004-05 and 41.6 in 2006-07 for the Reassessment component. 

In 2004-05, there were a total of 24 Part C monitoring processes; of these, all 24 included a 
Problem Identification component, 8 included a Problem Investigation component, 2 1 included a 
Corrective Action and Enforcement component, 1 1 included an Improvement Planning and 
Implementation component, and 12 included a Reassessment component. In 2006-07, there were 
a total of 28 Part C monitoring processes; of these, all 28 included a Problem Identification 
component, 8 included a Problem Investigation component, 23 included a Corrective Action and 
Enforcement component, 12 included an Improvement Planning and Implementation component, 
and 10 included a Reassessment component. The average component scores ranged from 42.9 
(Reassessment) to 75.3 (Corrective Action and Enforcement) in 2004-05 and from 34.3 
(Reassessment) to 73.8 (Corrective Action and Enforcement) in 2006-07. More specifically, the 
average estimated percentage of elements present for those Part C monitoring processes that had 
each component was as follows: 

• 59.0 in 2004-05 and 68.8 in 2006-07 for the Problem Identification component; 

• 60.1 in 2004-05 and 63.4 in 2006-07 for the Problem Investigation component; 

• 75.3 in 2004-05 and 73.8 in 2006-07 for the Corrective Action and Enforcement 
component; 

• 57.3 in 2004-05 and 54.1 in 2006-07 for the Improvement Planning and 
Implementation component; and 

• 42.9 in 2004-05 and 34.3 in 2006-07 for the Reassessment component. 

Individual Elements and Part B and Part C Monitoring Processes 

Next, the report examined the individual elements associated with framework components and 
the number and proportion of state Part B and Part C monitoring processes in which each 
element was present. The analyses focused on the elements that were present most and least 
often in the Part B or Part C monitoring processes that had the component, with most frequently 
being defined as those elements that were present in two-thirds or more (>67%) of the processes 
and least frequently being defined as those elements that were present in than one-third (<33%) 
of the processes. 
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The individual elements that were most often present in Part B monitoring processes in either 
time period included the following: 

• timelines included for each step/component of CAPs (E1B2, 97% in 2004-05); 

• opportunities for LEA/EIS programs to discuss improvement plans with SEA/lead 
agency staff (F4B, 96% in 2004-05; 95% in 2006-07); 

• informing state leadership of the problem investigation process (D1C, 94% in 2006- 
07); 

• training for individuals conducting problem investigation (D2B, 94% in both years); 
and 

• documentation of purposes for collecting problem investigation data (D4B1, 94% in 
both years). 

The individual elements that were least often present in Part B monitoring processes in either 
time period included the following: 

• documentation of evidence-based practices for improvement planning (F3A2, 5% in 
2006-07); 

• dissemination of CAPs to local stakeholders beyond LEA/EIS personnel (E2A, 9% in 
2004-05; 13% in 2006-07); and 

• dissemination of reassessment reports to local stakeholders other than LEA/EIS 
personnel (H3A, 1 1% in 2006-07). 

For Part B, 65 elements were used in both rounds of site visits. Overall, 26 of the 65 elements 
(40%) in 2004-05 and 34 of the 65 elements (52%) in 2006-07 were present in two-thirds or 
more of the Part B monitoring processes. At the other end of the spectrum, 9 of the 65 elements 
(14%) in 2004-05 and 8 of the 65 elements (12%) in 2006-07 were present in less than one -third 
of the Part B monitoring processes. 

The individual elements that were most often present in Part C monitoring processes in either 
time period included the following: 

• training for individuals conducting problem investigation (D2B, 100% in 2006-07); 

• specification of problem areas in CAPs (E1A1, 100% in 2004-05); 

• opportunities for LEA/EIS programs to discuss CAPs with SEA/lead agency staff 
(E2B, 1 00% in 2004-05 ; 96% in 2006-07); 

• specification of problem areas in improvement plans (F2A1, 100% in 2004-05); 

• improvement plans addressing all problem areas (F2A2, 100% in 2004-05); and 
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• problem identification findings adequately supported by data (C1C, 93% in 2006-07). 

The individual elements that were least often present in Part C monitoring processes in either 
time period included the following: 

• dissemination of reassessment reports to local stakeholders other than LEA/EIS 
personnel (H3A, 0% in 2004-05; 10% in 2006-07); 

• opportunities for local stakeholders to discuss reassessment reports with SEA/lead 
agency staff (H3C, 0% in 2004-05); 

• specification of financial support for implementation in improvement plans (F3B2, 
8% in 2006-07); and 

• documentation of research and evidence-based practices in improvement plans 
(F3A2, 9% in 2004-05; 0% in 2006-07). 

For Part C, 64 elements were used in both rounds of site visits. Overall, 25 of the 64 elements 
(39%) in 2004-05 and 28 of the 64 elements (44%) in 2006-07 were present in two-thirds or 
more of the Part C monitoring processes. On the other end of the spectrum, 13 of the 64 elements 
(20%) in 2004-05 and 8 of the 64 elements (13%) in 2006-07 were present in less than one -third 
of the Part C monitoring processes. 



Conclusions 

This study was designed to examine states’ Part B and Part C monitoring and improvement 
practices under IDEA. Although the variation in state monitoring practices has been noted 
previously (e.g., Comstock-Galagan and O’Connell 2002; Rostetter 1988), this study was the 
first to conduct an independent and systematic examination of Part B and Part C monitoring 
systems across a sample of states. This examination was achieved through two rounds of in- 
depth site visits to a systematic random sample of 20 states, in which the project team looked at 
the Part B and Part C monitoring systems that were in place in 2004-05 and 2006-07 using a 
framework for state monitoring developed for this study. As discussed in chapter 3, the 
framework included components, key characteristics, and elements. A component represented a 
set of distinct and coherent tasks, each with its own unique purpose and focus related to 
monitoring implementation of IDEA; key characteristics defined the components and identified 
aspects thought to be important to the component being carried out; and the elements identified 
what signified the presence of each key characteristic. State monitoring systems could consist of 
one or more distinct processes, each of which could differ in how they mapped onto the 
framework. 

Consistent with previous literature, the study found substantial variability in the nature and 
design of states’ monitoring systems in both 2004-05 and 2006-07. Findings indicated that states’ 
monitoring systems are often changing and evolving, and also that states approach monitoring 
activities in diverse ways. For example, as described in the findings (see chapter 5) and in the 
summary (see chapter 6), in both time periods, a percentage of states indicated that their 



116 




monitoring system was in transition (45% of the Part B and Part C monitoring systems in 2004- 
OS and 60% of the Part B monitoring systems and 35% of the Part C monitoring systems in 
2006-07). Although the individual circumstances varied, some of these states were phasing out 
existing monitoring processes and replacing them with new or revised processes. Other states 
were implementing new monitoring processes to complement existing processes. In addition, 
states also varied in the number of processes that made up their monitoring systems. For Part B, 
half of the state monitoring systems were composed of a single process, whereas the rest were 
composed of multiple processes in both years. For Part C, in both 2004-05 and 2006-07, the 
majority of state monitoring systems were composed of a single process (17 in 2004-05 and 13 in 
2006-07). States’ Part B and Part C monitoring systems also varied in the extent to which their 
processes included components from the framework. Few Part B and Part C monitoring 
processes included all five components (for Part B, four processes for both 2004-05 and 2006- 
07; for Part C, three processes in 2004-05 and one process in 2006-07). In addition, states varied 
in how they carried out each framework component (e.g., how they selected indicators and 
targets, the sources of data they used to identify and investigate problems, how they chose to 
address identified problems). 

According to IDEA, the overarching purposes of states’ monitoring systems are to improve the 
educational and functional outcomes for all children with disabilities and to ensure compliance 
with IDEA requirements, with a particular emphasis on those requirements that are most closely 
related to improving educational results for children with disabilities. As discussed throughout 
this report, many factors can potentially influence states’ monitoring systems, including changes 
to federal and state laws and regulations, guidance from OSEP, technical assistance received 
from various sources (e.g., NCSEAM), and contextual factors (e.g., state educational policies). 
The variation in the nature and design of states Part B and Part C monitoring systems, in 
combination with the change and evolution of these systems from year-to-year, makes it 
challenging to attempt to detennine what features of state monitoring systems will lead to 
improved compliance with IDEA and better outcomes for children and students with disabilities. 

The framework that guided the site visit data collection was developed for organizing the 
description of state monitoring systems and ensuring that important details about these systems 
were systematically documented. The framework enabled the project team to examine such 
topics as: (1) which components states included in their Part B and Part C monitoring processes 
(e.g., How many components did states monitoring processes have? How did they combine these 
components?); (2) how states approached each of the components (e.g., How did they select their 
indicators and targets? What kinds of data sources did they use to identify and investigate 
problems? How did they address identified problems?); and (3) which elements were most and 
least often present in states’ Part B and Part C monitoring processes (e.g., Do they have 
measureable indicators (A3B) and documented targets (A4B)? Are findings adequately 
supported by data (C1C)?). 

Although the framework successfully served its purpose in that it allowed the project team to 
systematically describe the variation that exists in states’ Part B and Part C monitoring systems, 
as noted earlier, it has not been validated or evaluated as to whether the adoption of this 
framework results in an improved state monitoring system. The conclusions from the findings of 
the study, therefore, are limited accordingly. For instance, the current study cannot speak to such 
issues as whether it is better for children, students, and family outcomes for states’ monitoring 
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processes to include all of the framework components as opposed to just some of them, if it is 
advantageous to approach certain components in one manner versus another, or whether the 
presence of a greater number of elements from the framework is associated with improved 
compliance or better outcomes for children and youth with disabilities. Although it would be 
challenging to evaluate the framework, or alternative frameworks given the current variation in 
states’ Part B and Part C monitoring systems, the validation of the framework used in this study 
or others would make a valuable contribution to the existing research on state monitoring 
systems. 
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Appendix A 
List of Acronyms 



APR - Annual Perfonnance Report 

BEH - U.S. Office of Education, Bureau of Education for the Handicapped 
CAP - corrective action plan 

CIFMS - Continuous Improvement Focused Monitoring System 

CIMP - Continuous Improvement Monitoring Process 

ED - U.S. Department of Education 

EHA - Education for all Handicapped Children Act 

EIS - early intervention services 

FAPE - free appropriate public education 

GEPA - General Education Provisions Act 

GPC - generalized partial credit 

ICC - intraclass correlation coefficient 

IDEA - Individuals with Disabilities Education Improvement Act of 2004 

IEP - individualized education program 

IES - Institute of Education Sciences 

IFSP - individualized family service plan 

IRT - item response theory 

JCSEE - Joint Committee on Standards for Educational Evaluations 

LEA - local education agency 

LRE - least restrictive environment 

MIS - management information system 

MSIP - Monitoring and State Improvement Planning Division 

NCD - National Council on Disability 

NCSEAM - National Center for Special Education Accountability Monitoring 

NCSER - National Center for Special Education Research 

OSEP -Office of Special Education Programs 

OSERS - Office of Special Education and Rehabilitative Services 

PL - parameter logistic 

SEA - state education agency 

SPP - State Performance Plan 
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Appendix B 

Framework Key Characteristics and Elements 



As described in chapter 3, Development of a Framework for State Monitoring Systems, key 
characteristics and elements were developed for each of the five components of the framework 
for state monitoring. In sum, a component represents a set of distinct and coherent tasks, each 
with its own unique purpose and focus related to monitoring implementation of the Individuals 
with Disabilities Education Act (IDEA); key characteristics define the components and identify 
aspects that are thought to be important to the component being carried out; and the elements 
spell out what signifies the presence of each key characteristic (see chapter 3 for more 
information about the framework). 

For example: 

Component: Problem Identification 

Key characteristic: A 1. Stakeholder Participation 

Stakeholders are informed and contribute throughout the indicator 
and target setting process. 

Element: A1A. Representatives from key constituencies (including advocacy 

groups, parents of children with disabilities, teachers, service 
providers, local education agency (LEA)/early intervention 
services (EIS) program and school administrators, leadership 
figures in special education and early intervention) participate in 
the process. 

This appendix presents the full set of key characteristics and elements that were used during the 
first round of data collection. After the first round of data collection, the number of key 
characteristics and elements was reduced using a Rasch analysis approach (see chapter 4 and 
appendix G for more information). The subset of key characteristics and elements that were used 
during the second round of site visits are denoted with an asterisk (*). 
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Component: Problem Identification 



A. Indicator and Target Setting 

At. Stakeholder Participation 

Stakeholders are informed and contribute throughout the indicator and target 

setting process. 

a. Representatives from key constituencies (including advocacy groups, parents 
of children with disabilities, teachers, service providers, local education 
agency (LEA)/early intervention services (EIS) program and school 
administrators, leadership figures in special education and early intervention) 
participate in the process. 

1. Stakeholder committee includes parents of children with disabilities or 
representatives from advocacy groups.* 

2. Parents of children with disabilities or the representatives from advocacy 
groups represent the diversity that exists in the state.* 

3. Materials for stakeholders are tailored to improve understanding for those 
who need it to permit their informed participation in the process.* 

4. Meetings are scheduled (time and location) to accommodate parents or 
representatives of advocacy groups. 

5. Stakeholder committee includes direct service personnel (i.e., teachers, 
service providers, and principals).* 

6. Stakeholder committee includes LEA/EIS program administrators (i.e., 
superintendents, special education/early intervention supervisors, other 
central office staff). 

7. Stakeholder committee includes a state leadership figure from special 
education or early intervention (i.e., state director of special education or 
Part C lead agency coordinator). 

b. State agency leadership figures in general education (i.e., state superintendent, 
state school board members) are informed (aware) of selected indicators and 
targets.* 

c. Leadership figures in key governmental positions (i.e., representative from the 
governor’s office, representative from the education or health committee of 
the state legislature) are informed (aware) of selected indicators and targets.* 

d. Both Part B and Part C program leadership figures are informed (aware) of 
selected indicators and targets. 

1. IF EVALUATING A PART B PROGRAM: Part C leadership figures are 
informed (aware) of selected indicators and targets used to monitor Part B. 

2. IF EVALUATING A PART C PROGRAM: Part B leadership figures are 
informed (aware) of selected indicators and targets used to monitor Part C. 
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A2. Indicator Identification 

There is a process to identify indicators. 

a. Indicators are based on stakeholder input. 

b. Indicator selection is based on a systematic process. 

c. Some of the indicators are outcome indicators. 

A3. Measurable Indicators 

Indicators are clearly written, logical, quantifiable, and represent valid measures 
of IDEA implementation. 

a. Indicator definitions are clear and unambiguous.* 

b. Indicators are measurable.* 

A4. Rigorous and Realistic Targets 

Selected targets are documented and reflect the state’s indicators. 

a. Target setting is accomplished through a systematic process.* 

b. Targets are documented.* 

c. Targets are reported as ranges, or as parts of a trend line. 

d. Targets reflect the indicators. 

Indicator Data Collection and Analysis 

Bl. Credibility of Data Collection Team(s) 

The individuals responsible for collecting the data used to measure LEA/EIS 
program performance on indicators possess the requisite technical competence, 
substantive knowledge, and experience. 

a. The indicator data collection process is managed by individuals at the state- 
level with relevant knowledge and experience. 

1. Persons responsible for managing data collection at the state-level are 
knowledgeable about data collection and database management. 

2. Persons responsible for managing data collection at the state-level are 
knowledgeable about monitoring and content issues related to the data. 

b. All data collection staff (state or local) is trained in each data collection effort. 

B2. Protection of Stakeholders 

The rights and welfare of those affected by the data collected to measure 
LEA/EIS program performance on indicators (i.e., infants, children and youth 
with disabilities and their families, teachers, and other services providers) are 
protected. 




a. The SEA/lead agency has written documentation that addresses protection of 
human subjects during data collection processes. 

b. Confidentiality agreements are signed by all data collectors and maintained by 
the SEA/lead agency. 

c. Written procedures for reporting incompetence, unethical behavior, fraud, 
waste, or abuse associated with data collection activities exist. 

d. Data collection managers and data collectors are trained in issues and methods 
related to the protection of human subjects. 

e. Confidentiality is supported through systematically stored and controlled data 
(access to which is governed by protocol and verifiable protection 
procedures). 

1 . Data are systematically stored. 

2. Access to confidential data is restricted. 

B3, Thoroughly Documented Data Collection Process 

The process for indicator data collection is described in sufficient detail so that 
the utility, reliability, and validity of the infonnation collected can be assessed. 

a. Data collection procedures and data sources are documented. 

1. Written documentation exists describing the purpose for collecting the 
data. 

2. Written documentation exists describing the data collection 
methodologies, including site visit plans (if indicator data collection 
involves site visits to LEAs/EIS programs).* 

3. Written documentation exists describing the sources from which data will 
be collected. 

b. Written documentation exists describing the sampling strategy used to collect 
indicator data. 

B4. Accurate Data 

Data collection instruments and procedures are reliable and valid for the purposes 
intended. The data collected are examined to ensure that errors are corrected or 
nonexistent. 

a. Procedures for coding and scoring data are documented.* 

b. Data collection procedures were pilot tested to identify and control sources of 
error. 

c. Procedures are in place to monitor the quality of the data collected.* 

d. Data provided directly to the state by LEAs/EIS programs are verified for 
accuracy.* 
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B5. Analysis of Indicator Data 

Indicator data are analyzed to detennine LEA/EIS program performance. 

a. Preliminary analyses are conducted to gain a thorough understanding of the 
indicator data. 

b. Analyses are conducted to assess performance on the indicators/targets.* 

B6. Practical and Efficient Procedures 

Indicator data collection procedures are practical and minimize burden on the 
LEA/EIS program, including limiting disruption of daily routines of schools or 
programs. 

a. No extraneous data are collected (i.e., all data collected are used to measure 
performance on the selected indicators). 

b. Procedures are in place to minimize interruptions/disruptions to school, 
program, or district routines when collecting data. 

C. Problem Detection 

Cl. Defensible Findings 

The findings drawn from the problem identification process are adequately 
defended and supported by comparing LEA/EIS program performance to 
specified targets. 

a. Findings reflect perfonnance in relation to specific targets.* 

b. Findings are limited to specific time periods. 

c. Findings are adequately supported by data.* 

C2. Complete Reporting on the State 

A report completely describes the problems that are identified throughout the state 
regarding perfonnance on specified indicators and targets, so that essential 
information can be provided and is easily accessible to stakeholders. 

a. State reports focus on the specified indicators and targets (i.e., discusses 
performance in relation to targets).* 

b. State reports describe the data used to assess performance on specified 
indicators and targets. 

c. State reports describe the methods and procedures used to identify problems.* 

d. State reports describe limitations of the data used to identify problems. 

e. Visual displays in state reports are clear and understandable and relate to the 
findings/data. 

f. State reports describe the extent of the problem in the state. 

g. Reports include a discussion of strengths and areas needing improvement. 
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h. Someone who did not write the report or collect the data reviews the report. 



C3. Complete Reporting on LEA/EIS Programs 

Reports prepared by the state completely describe the problems that are identified 

regarding LEA/EIS program performance on specified indicators and targets, so 

that essential information can be provided and is easily accessible to local 

stakeholders. 

a. Local-level reports focus on the specified indicators and targets (i.e., discusses 
performance in relation to targets).* 

b. Local-level reports describe the data used to assess perfonnance on specified 
indicators and targets. 

c. Local-level reports describe the methods and procedures used to identify 
problems. 

d. Local-level reports describe limitations of the data used to identify problems. 

e. Visual displays in local-level reports are clear and understandable and relate 
to the findings/data. 

f. Reports on LEAs/EIS programs include a discussion of strengths and areas 
needing improvement. 

g. Someone who did not write the report or collect the data reviews the report. 

C4. Dissemination of Problem Identification Reports to Stakeholders 

Problem identification reports are accessible and available to stakeholders. 

a. State-level problem identification reports are disseminated to stakeholders.* 

b. Local level problem identification reports are disseminated to local 
stakeholders beyond the LEA/EIS program administrators.* 

c. Opportunities are provided for stakeholders to discuss state-level problem 
identification reports with SEA/lead agency staff* 

d. Opportunities are provided for LEA/EIS program personnel to discuss local- 
level problem identification reports with SEA/lead agency staff. 

e. Opportunities are provided for local stakeholders to discuss local-level 
problem identification reports with SEA/lead agency staff. 



Component: Problem Investigation 

D. Problem Investigation 

Dl. Stakeholder Participation 

Stakeholders are informed and contribute throughout the problem investigation 
process. 
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a. Representatives from key constituencies (including advocacy groups, parents 
of children with disabilities, teachers, service providers, LEA/EIS program 
and school administrators) participate in the process. 

1 . Input is sought from parents of children with disabilities or representatives 
from advocacy groups.* 

2. Parents of children with disabilities or the representatives from advocacy 
groups represent the diversity that exists in the state. 

3. Materials used to seek input from stakeholders are tailored to improve 
understanding and facilitate their contribution to the problem investigation 
process.* 

4. Input is sought from direct service personnel (i.e., teachers, service 
providers, and principals). 

b. LEA/EIS program administrators (e.g., superintendents, special education/ 
early intervention supervisors) contribute to the problem investigation 
process. 

c. A state leadership figure from special education or early intervention (e.g., 
state director of special education, Part C lead agency coordinator, the state 
monitoring director) are informed (are aware) of the problem investigation 
process.* 

D2. Credibility of Problem Investigation Team 

The individuals responsible for investigating identified problems possess the 
requisite technical competence, substantive knowledge, and experience. 

a. Managers (of problem investigations) have a background and knowledge 
related to identified problems. 

b. Individuals conducting problem investigations have appropriate training.* 

D3. Practical and Efficient Procedures 

Data collection procedures for problem investigations are practical, internally 
consistent, and efficient while minimizing burdens on the LEA/EIS program, 
including the disruption of daily routines of schools or programs. 

a. No extraneous data are collected (i.e., all data collected are used to investigate 
identified problems). 

b. Procedures are in place to minimize interruptions/disruptions to school, 
program, or district routines when collecting data.* 

D4. Thoroughly Documented Problem Investigations 

The process used to investigate identified problems is described in sufficient 
detail so that the utility, reliability, and validity of the information can be 
assessed. 

a. Each problem being investigated is clearly defined and documented. 
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b. Data collection methodologies and data sources are documented. 

1. Written documentation exists describing the purpose for collecting the 
data.* 

2. Written documentation describes data collection methodologies, including 
site visit plans (if problem investigations involve site visits to LEAs/EIS 
programs). 

c. Written documentation exists describing the sources from which data will be 
collected. 

d. Written documentation exists describing the sampling strategy used to 
investigate identified problems. 

D5. Accurate Data 

Data collection instruments and procedures are reliable and valid for the purposes 
intended. The data collected are examined to ensure that errors are corrected or 
nonexistent. 

a. Procedures for coding and scoring data are documented.* 

b. Procedures are in place to monitor the quality of the data collected.* 

c. Data are verified for accuracy.* 

D6. Analysis of Quantitative Data 

The analytic techniques fit the quantitative data and the identified problems under 
investigation. 

a. Exploratory analyses are conducted to ensure the data’s correctness and to 
gain an increased understanding of the data. 

b. Data variability is examined.* 

c. Statistics presented in reports or presentations reflect the nature of the data 
and the identified problems under investigation. 

D7. Analysis of Qualitative Data 

The analytic techniques fit the qualitative data and the identified problems under 
investigation. 

a. Multiple methodologies are used to verify the accuracy of findings (e.g., 
triangulation). 

b. Analytic procedures and methods of summarization are used. 

D8. Defensible Findings 

The findings from the problem investigation process are adequately documented, 
defended, and supported by data, so that the state can proceed with improvement 
planning or corrective actions. 

a. Findings focus directly on the identified problems under investigation.* 
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b. Findings and conclusions are adequately supported by data. 



Component: Corrective Action and Enforcement 

E. Corrective Action and Enforcement 

El. Detailed Corrective Action Plans 

Corrective action plans (CAPs) address the identified problem(s) related to 
procedural noncompliance and are sufficiently detailed. 

a. LEA/EIS program CAPs address the identified problem(s) related to 
procedural noncompliance. 

1 . CAPs specify all problem areas.* 

2. Written action plans address each problem area. 

b. LEA/EIS program CAPs are detailed and include steps/components and 
timelines. 

1. Steps/components of the CAPs are clearly delineated for each identified 
problem.* 

2. Timelines are included for each step/component of the CAPs.* 

3. Plans to measure progress toward meeting timelines are documented. 

E2. Dissemination of Corrective Action Plans to Stakeholders 

Corrective action plans (CAPs) are accessible and available to local stakeholders. 

a. CAPs are disseminated to local stakeholders beyond the LEA/EIS program 
administrators.* 

b. Opportunities are provided for LEA/EIS program personnel to discuss CAPs 
with SEA/lead agency staff.* 

c. Opportunities are provided to local stakeholders to discuss CAPs with 
SEA/lead agency staff. 

E3. Enforcement of Corrective Action Plans 

Enforcement procedures are in place to ensure compliance with needed corrective 
action. 

a. Enforcement procedures are documented. 

1 . The SEA/lead agency has written general enforcement procedures.* 

2. General enforcement procedures delineate specific actions that will be 
taken by the SEA/lead agency if a CAP is not implemented.* 

b. General enforcement procedures are accessible to stakeholders. 
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Component: Improvement Planning and Implementation 



F. Improvement Planning 

FI. Stakeholder Participation 

Stakeholders are informed and contribute throughout the improvement planning 
process. 

a. Representatives from key constituencies (including advocacy groups, parents 
of children with disabilities, teachers, service providers, LEA/EIS program 
and school administrators) participate in the process. 

1 . Input is sought from parents of children with disabilities or representatives 
from advocacy groups.* 

2. Parents of children with disabilities or the representatives from advocacy 
groups represent the diversity that exists in the state. 

3. Materials used to seek input from stakeholders are tailored to improve 
understanding and facilitate their contribution to the improvement 
planning process.* 

4. Input is sought from direct service personnel (i.e., teachers, service 
providers, and principals). 

b. LEA/EIS program administrators (e.g., superintendents, special education/ 
early intervention supervisors) contribute to the improvement planning 
process. 

c. A state leadership figure from special education or early intervention (e.g., 
state director of special education, Part C lead agency coordinator, the state 
monitoring director) is informed (is aware) of the improvement planning 
process. 

F2. Detailed Improvement Plans 

Improvement plans address the identified problems and are sufficiently detailed. 

a. LEA/EIS program improvement plans address identified problems. 

1 . Improvement plans specify all problem areas.* 

2. Written improvement plans address each problem area.* 

b. Improvement plans include steps/components and timelines. 

1. Steps/components of the improvement plans are clearly delineated for 
each identified problem.* 

2. Timelines are included for each step/component of the improvement plans. 

c. Improvement plans include a description of data or information to be used to 
evaluate success of the plan. 
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F3. Defensible Improvement Plans 

Improvement plans reflect research and evidence-based practices, identify support 
for implementation, and include oversight. 

a. Improvement plans include properly cited supporting infonnation and data 
(e.g., evidence-based practices). 

1 . Improvement plans are supported by data from the problem investigation 
process. 

2. Improvement plans include documentation of research and evidence-based 
practices on which the plans are based.* 

b. Plans outline intended support for LEAs/EIS programs. 

1. Plans specify TA to support implementation.* 

2. The plan specifies financial support for implementation.* 

c. Plans describe the ways that implementation will be monitored by the 
SEA/lead agency.* 

F4. Dissemination of Improvement Plans to Stakeholders 

Improvement plans are accessible and available to stakeholders. 

a. Improvement plans are disseminated to local stakeholders beyond the 
LEA/EIS program administrators.* 

b. Opportunities are provided for LEA/EIS program personnel to discuss 
improvement plans with SEA/lead agency staff.* 

c. Opportunities are provided for local stakeholders to discuss improvement 
plans with SEA/lead agency staff.* 

G. Improvement Plan Implementation 

Gl. Plans and Resources for Implementation 

Resources and support are provided for implementation of the improvement plan. 

a. SEA/lead agency meets with LEAs/EIS programs to help put the improvement 
plan into action. 

1. Meetings are held with SEA/lead agency to discuss local implementation.* 

2. LEAs/EIS programs provide regular updates to the SEA/lead agency on 
implementation progress (report, letter, memo, meeting, conference call, 
etc). 

b. SEA/lead agency provides resources to LEAs/EIS programs that allow them 
to obtain additional technical assistance, training, supplemental data, funding, 
etc. related to improvement plan implementation. 
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G2. Fidelity of Implementation 

Plans are fully and evenly implemented in LEAs/EIS programs. 

a. Documentation confirms that plans were followed or changes in plans 
justified.* 

b. SEA/lead agency provides follow-up and feedback to LEAs/EIS programs on 
a regular basis concerning their progress in implementing improvement 
plans.* 

Component: Reassessment 

H. Reassessment 

HI. Defensible Findings 

The findings drawn from the reassessment process are adequately defended and 

supported by comparing LEA/EIS program performance to specified targets. 

a. Findings reflect performance in relation to specific targets. 

b. Findings are adequately supported by data. 

H2. Complete Reporting 

Reassessment reports completely describe the reassessment process so that 

essential information is provided and easily accessible to LEAs/EIS programs and 

other stakeholders. 

a. Reports focus on specified indicators and targets.* 

b. Reports describe the data used to reassess performance/compliance on 
specified indicators and targets.* 

c. Reports describe the methods and procedures used to reassess performance or 
compliance.* 

d. Reports describe limitations of the data used to reassess performance or 
compliance. Visual displays are clear and understandable and relate to the 
findings/data. 

e. Visual displays are clear and understandable and relate to the findings/data. 

f. Reassessments of LEA/EIS programs include a discussion of strengths and 
areas needing further improvement.* 

g. Someone who did not write the report or collect the data reviews the reports.* 
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H3. Dissemination of Reassessment Reports to Stakeholders 

Reassessment reports are accessible and available to stakeholders. 

a. Reassessment reports are disseminated to local stakeholders beyond the 
LEA/EIS program administrators.* 

b. Opportunities are provided for LEA/EIS program personnel to discuss 
reassessment reports with SEA/lead agency staff.* 

c. Opportunities are provided for local stakeholders to discuss reassessment 
reports with SEA/lead agency staff* 

d. Findings from reassessments are disseminated to state-level stakeholders.* 
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Appendix C 

Element Rating Instructions 



This appendix presents the element rating instructions that the project team provided to site 
visitors to assist them with making their ratings. The rating instructions are organized by 
component, with the key characteristics and elements listed under each component. Specific 
rating instructions are presented for each element. Examples are provided throughout that helped 
further clarify to site visitors how to make ratings. 



Component: Problem Identification 

A. Indicator and Target Setting 

At. Stakeholder Participation 

Stakeholders are informed and contribute throughout the indicator- and target- 
setting process. 

General Guideline: This key characteristic refers to stakeholder participation at 
the state-level. The stakeholders can belong to an existing committee that 
dedicates time to this purpose or belong to a committee convened especially for 
this purpose. For performance/outcomes indicators, stakeholders should be 
involved in the process used to set the indicators and targets the state uses to 
monitor local districts and programs. For compliance/procedural indicators, 
stakeholders shoidd be involved in the process used to set the indicators the state 
uses to monitor compliance with IDEA procedural requirements. 

When interviewing the monitoring director 1 and gathering information on the 
state ’.s' indicator and target setting process, assess direct stakeholder involvement 
in this process (e.g., is indicator and target setting an agenda item at the 
stakeholder committee meeting?). Review a written list of stakeholders to 
determine represen tation. If the written list is not clear, ask the interviewee about 
specific stakeholders noted below. Review any written documentation of 
stakeholder involvement (i.e., memos, meeting minutes, copies of materials/ 
reports distributed to members), but written documentation is not required to rate 
the following elements. Interview a member of the stakeholder committee who is a 
parent or advocate before finalizing your ratings. 

a. Representatives from key constituencies (including advocacy groups, parents 
of children with disabilities, teachers, service providers, local education 
agency (LEA)/early intervention services (EIS) program and school 
administrators, leadership figures in special education and early intervention) 
participate in the process. 



1 The term monitoring director is generally used throughout this appendix to refer the person who leads the state’s 
Part B or Part C monitoring effort (i.e., state director of special education, state monitoring coordinator, Part C 
coordinator). 
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1. Stakeholder committee includes parents of children with disabilities or 
representatives from advocacy groups. [YES/NO] 

For this rating to be a “YES, ” the monitoring director or written 
documentation must indicate that the stakeholder committee includes at 
least two parents of children with disabilities or disability advocates. 
These stakeholders should represent different disability categories. 

2. Parents of children with disabilities or the representatives from advocacy 
groups represent the diversity that exists in the state. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe how 
state diversity is represented on the committee. This will vary from state- 
to-state depending on the demographics of the state. Some examples are 
representatives from rural and urban areas or representatives from 
different racial/ethnic groups. 

3. Materials for stakeholders are tailored to improve understanding for those 
who need it to permit their informed participation in the process. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for providing additional background information and explaining 
technical terms/jargon to stakeholders. Such a process might include a 
review of materials by someone who is expected to think about potential 
problems with readability or accessibility of the information. If the 
monitoring director indicates that this is not necessary for the stakeholder 
committee or that nothing special is done to tailor materials, then the 
rating would be a “NO. ” 

4. Meetings are scheduled (time and location) to accommodate parents or 
representatives of advocacy groups. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe how 
meeting times and locations are planned to address different schedules 
and places of residence. 

5. Stakeholder committee includes direct service personnel (i.e., teachers, 
service providers, and principals). [YES/NO] 

For this rating to be a “YES, ” the monitoring director or written 
documentation must indicate that the stakeholder committee includes at 
least two direct service personnel. 

6. Stakeholder committee includes LEA/EIS program administrators (i.e., 
superintendents, special education/early intervention supervisors, other 
central office staff). [YES/NO] 

For this rating to be a “YES, ’’ the monitoring director or written 
documentation must indicate that the stakeholder committee includes at 
least two program administrators. 



C-2 




7. Stakeholder committee includes a state leadership figure from special 
education or early intervention (i.e., state director of special education or 
Part C lead agency coordinator). [YES/NO] 

For this rating to be a “YES, ” the monitoring director or written 
documentation must indicate that the stakeholder committee includes a 
state leadership figure. 

b. State agency leadership figures in general education (i.e., state superintendent, 
state school board members) are informed (aware) of selected indicators and 
targets. [YES/NO/NA] 

For this rating to be a “YES, ” the monitoring director must describe a 
specific process for keeping these general education leadership figures 
informed, which may include scheduling time on a regular meeting agenda, 
sending a report, or holding a conference call. 

This element should be rated “NA ” if evaluating Part C. 

c. Leadership figures in key governmental positions (i.e., representative from the 
governor’s office, representative from the education or health committee of 
the state legislature) are infonned (aware) of selected indicators and targets. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
specific process for keeping these governmental leaders informed, which may 
include scheduling time on a regular meeting agenda, sending a report, or 
holding a conference call. 

d. Both Part B and Part C program leadership figures are informed (aware) of 
selected indicators and targets. 

1. IF EVALUATING A PART B PROGRAM: Part C leadership figures are 
informed (aware) of selected indicators and targets used to monitor Part B. 
[YES/NO/NA] 

For this rating to be a “YES, ” the monitoring director must describe a 
specific process for keeping these stakeholders informed, which may 
include scheduling time on a regular meeting agenda, sending a 
report, or holding a conference call. 

This element should be rated “NA ” if evaluating Part C. 

2. IF EVALUATING A PART C PROGRAM: Part B leadership figures are 
informed (aware) of selected indicators and targets used to monitor Part C. 
[YES/NO/NA] 

For this rating to be a “YES, ” the monitoring director must describe a 
specific process for keeping these stakeholders informed, which may 
include scheduling time on a regular meeting agenda, sending a 
report, or holding a conference call. 

This element should be rated “NA ” for evaluating Part B. 
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A2. Indicator Identification 



There is a process to identify indicators. 

General Guidelines: Indicators refer to the measures the state uses to examine 
compliance or outcome performance for the purposes of verifying success and 
iden tifying poten tial problems. This key characteristic concerns the quality of the 
process used to identify those specific measures (i.e., indicators) the state will use 
to monitor LEAs/EIS programs. 

When interviewing the monitoring director, determine how the state decided upon 
the indicators it uses to monitor LEAs/EIS programs (i.e., how did the indicators 
come to be?). Review any written documentation (i.e., reports or memos) 
describing the state’s indicator selection process; however, written 
documentation is not required for rating the following elements. 

a. Indicators are based on stakeholder input. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for obtaining stakeholder input on the selection of indicators. For 
example, this could include a series of meetings or conference calls to discuss 
possible indicators or soliciting feedback on written materials (including 
surveys). 

b. Indicator selection is based on a systematic process. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for selecting indicators. The process must involve multiple steps with 
review and consideration of proposed indicators at each step. 

c. Some of the indicators are outcome indicators. [YES/NO] 

For this rating to be a “YES, ” at least one indicator the state uses to monitor 
LEAs/EIS programs focuses on outcomes for children with disabilities or their 
families (e.g., graduation/dropout rates, performance on statewide 
assessments, developmental or functional abilities of infants and toddlers). 

A3. Measurable Indicators 

Indicators are clearly written, logical, quantifiable, and represent valid measures 
of IDEA implementation. 

General Guidelines: Review documentation of the indicators the state uses to 
monitor LEAs/EIS programs, including definitions. Written documentation is 
required to rate the following elements. If no written documentation exists, then 
the rating is a “0. ” 

a. Indicator definitions are clear and unambiguous. [0/1/2] 

Clear and unambiguous refers to indicators that are understandable and have 
no competing interpretation. The rating is “2 ” if ALL indicator definitions are 
understandable and have no competing interpretation; the rating is “ 1 ” if 
only SOME indicator definitions are understandable and have no competing 
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interpretation; the rating is a “0” if NO indicator definitions are 
understandable and have no competing interpretation. 

b. Indicators are measurable. [0/1/2] 

Measurable means that indicators are quantifiable. For example, children 
exiting Part C programs are enrolled in preschool programs or students 
exiting Part B are enrolled in post-secondary education programs. The rating 
is “. 2 ” if ALL indicators are measurable; the rating is “1 ” if only SOME 
indicators are measurable; the rating is “0” if NO indicators are measurable. 

A4. Rigorous and Realistic Targets 

Selected targets are documented and reflect the state’s indicators. 

General Guidelines: This key characteristic does not apply to compliance 
indicators. Compliance indicators have an implied target of 0% or 100%. Rate 
this key characteristic for performance/outcomes indicators only. Do not rate A4 
if there are no outcome indicators. 

When interviewing the monitoring director, ask about the targets used to measure 
performance on the indicators and about the process for arriving at those targets 
(i.e., how did the targets come to be?). Review documentation ( i. e. , reports or 
memos) describing the state ’s targets. Written documentation is required, except 
where noted. 

a. Target setting is accomplished through a systematic process. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for setting targets with multiple steps with review and consideration 
of the proposed targets at each step. Written documentation is not required to 
rate this element. 

b. Targets are documented. [0/1/2] 

All targets are documented in meeting minutes, memos, reports, or other 
written format. The rating is “2” if ALL targets are documented; the rating is 
“1 ” if only SOME targets are documented; the rating is “0” if NO targets are 
documented. 

c. Targets are reported as ranges, or as parts of a trend line. [0/1/2] 

Targets should be reported as ranges (e.g., the target for graduation rates for 
children with disabilities is 80-95%) or as part of a trend line (e.g., 
graduation rates for children with disabilities increase by 10% each year). 
The rating is “2 ” if ALL targets are reported as ranges or part of a trend line; 
the rating is “1” if only SOME targets are reported as ranges or part of a 
trend line; the rating is “ 0 ” if NO targets are reported as ranges or part of a 
trend line. 
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d. Targets reflect the indicators. [0/1/2] 

The targets clearly match the corresponding indicators. For example, if the 
indicator is a comparison of special education suspension rates and non- 
special education suspension rates, and the target focuses only on special 
education suspension rates, the target does NOT reflect the indicator. The 
rating is “ 2 ” if ALL targets reflect the indicators; the rating is “1 ” if SOME 
targets reflect the indicators; the rating is “0” if NO targets reflect the 
indicators. 

B. Indicator Data Collection and Analysis 

Note: If indicator data collection and analysis involved data from statewide and other 
standardized assessments, assume these data are of high quality (i.e., accurate, valid, and 
reliable). States that used statewide and other standardized assessments in their indicator 
data collection and analysis cannot receive a rating of “0. ” If indicator data collection 
and analysis ONLY involved statewide and other standardized assessments, you would 
rate the relevant elements as “2. ” If the state used statewide and other standardized 
assessments AND other types of data, they must receive either a “. 2 ” (if the element is 
true for ALL the other data collections) or a “1 ” (if the element is true for SOME, but not 
ail, of the other data collections). 

With only a couple of exceptions, the ratings for most elements will use the interval scale 
(0/1/2) due to the likely possibility of separate data collection processes for each 
indicator. That is, there may be multiple data collections to take into account when 
making ratings for these elements. Therefore, the ratings for each element should reflect 
ALL data collection efforts. For example, multiple data collection efforts might include 
collection of data about (1) the number of students who receive 75% or more of their 
education in the general education setting; and (2) the dropout rates in local school 
districts. The following elements would apply to each of these efforts. 

Bl. Credibility of Data Collection Team(s) 

The individuals responsible for collecting the data used to measure LEA/EIS 
program performance on indicators possess the requisite technical competence, 
substantive knowledge, and experience. 

General Guidelines: When interviewing the monitoring director, determine who is 
responsible for collecting data the state uses to monitor LEAs/EIS programs. Ask 
about the experience and training of person(s) managing the data collection and 
persons collecting data. Review any documentation on training of data collection 
staff, but written documentation is not_ required to rate the following elements. If 
the monitoring director does not manage the data collection, you should also 
interview the data collection manager(s). The following elements concern 
multiple data collections. 

a. The indicator data collection process is managed by individuals at the state- 
level with relevant knowledge and experience. 
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1. Persons responsible for managing data collection at the state-level are 
knowledgeable about data collection and database management. [0/1/2] 

The person(s) managing each data collection effort has experience 
working with large quantitative data sets and understands issues related 
to “clean ” and valid data. The rating is “ 2 ” if managers of ALL data 
collections are knowledgeable; the rating is “1 ” if managers of SOME 
data collections are knowledgeable; the rating is “0” if managers of NO 
data collections are knowledgeable. 

2. Persons responsible for managing data collection at the state-level are 
knowledgeable about monitoring and content issues related to the data. 
[0/1/2] 

The person managing each data collection has an understanding of the 
state’s monitoring system and the indicators/targets the data are 
measuring. The rating is “ 2 ” if managers of ALL data collections are 
knowledgeable; the rating is “1 ” if managers of only SOME data 
collections are knowledgeable; the rating is “0 ” if managers of NO data 
collections are knowledgeable. 

b. All data collection staff (state or local) is trained in each data collection effort. 

[0/1/2] 

Staff members who collect the indicator and target data have at least one 
opportunity to receive training on how to collect reliable and valid data and 
they participated in the training. This includes, for example, local staff 
collecting self-assessment data, local staff collecting data sent to the state, 
state staff conducting desk audits, and state staff conducting site visits. Staff 
members should be trained in each data collection in which they are involved. 
The interviewee should be able to describe a specific training process or 
provide written training materials for you to review. The rating is “2 ” if staff 
members involved in EACH data collection are trained; the rating is “1 ” if 
staff members in volved in only SOME data collections are trained; the rating 
is “0” if NO data collection staff members are trained. 

B2. Protection of Stakeholders 

The rights and welfare of those affected by the data collected to measure 
LEA/EIS program performance on indicators (i.e., infants, children and youth 
with disabilities and their families, teachers, and other services providers) are 
protected. 

General Guidelines: When interviewing the monitoring director or data collection 
manager, ask about state policies and procedures to ensure confidentiality and 
protection of the rights of stakeholders with regard to the monitoring data 
collected. Review any documentation, such as policy manuals, memos, or public 
postings. Written documentation is required to rate the following elements, except 
where noted. 
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a. The SEA/lead agency has written documentation that addresses protection of 
human subjects during data collection processes. [YES/NO] 

For this rating to be a “YES, ” you must review documentation that outlines 
procedures for ensuring personal information is kept confidential and private. 
If this documentation was not written specifically for monitoring, it must be 
clear that it is applied to monitoring data collection processes. 

b. Confidentiality agreements are signed by all data collectors and maintained by 
the SEA/lead agency. [YES/NO] 

For this rating to be a “YES, ” the monitoring director or data collection 
manager must state that all data collectors — state and local level — sign 
confidentiality agreements and that these agreements are kept on file. You do 
not need to see the agreements to rate this element. 

c. Written procedures for reporting incompetence, unethical behavior, fraud, 
waste, or abuse associated with data collection activities exist. [YES/NO] 

For this rating to be a “YES, ” you must review documentation that specifies 
what steps staff should take if a data collector engages in this kind of activity. 
If this documentation was not written specifically for monitoring, it must be 
clear that it is applied to monitoring data collection processes. 

d. Data collection managers and data collectors are trained in issues and methods 
related to the protection of human subjects. [0/1/2] 

Staff who manage or collect the indicator and target data have at least one 
opportunity to receive training on procedures for ensuring personal 
information is kept confidential and private and they participated in the 
training. This includes both state and local staff members. The interviewee 
should be able to describe a specific formal training process or written 
training materials are provided for you to review. The rating is “2 ” if staff 
involved in EACH data collection effort receives formal training; the rating is 
“1 ” if staff involved in SOME data collection efforts receives formal training; 
the rating is “0 ” if NO staff receives formal trained in any data collection 
effort. You do not need to see written documentation to rate this element. 

e. Confidentiality is supported through systematically stored and controlled data 
(access to which is governed by protocol and verifiable protection 
procedures). 

1. Data are systematically stored. [0/1/2] 

The monitoring director or data collection manager must be able to 
describe specific procedures for storing monitoring data that are either in 
hard copy (paper) or electronic formats. Procedures must include storing 
and labeling the data and procedures for main taining confidentiality. The 
rating is “2 ” if data from ALL data collections are systematically stored; 
the rating is “ 1 ” if data from SOME data collections are systematically 
stored; the rating is “ 0 ” if data from NO data collections are 
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systematically stored. You do not need to see written documentation to 
rate this element. 

2. Access to confidential data is restricted. [0/1/2] 

The monitoring director or data collection manager must be able to 
describe how access to confidential data (e.g., names, addresses, social 
security numbers) is controlled. For example, data are stored on 
password-protected computers or in locked cabinets, and only staff with 
specific responsibilities related to the data may gain entry. The rating is 
“2 ” if access to confidential data is restricted for ALL data collections; 
the rating is “1 ” if access to confidential data is restricted for only SOME 
data collections; the rating is “0” if access to confidential data is NOT 
restricted for any data collections. You do not need to see written 
documentation to rate this element. 

B3. Thoroughly Documented Data Collection Process 

The process for indicator data collection is described in sufficient detail so that 
the utility, reliability, and validity of the infonnation collected can be assessed. 

General Guidelines: Review documentation (i.e., user manuals, reports, memos) 
related to the indicator data collection process. Clarify that the documentation 
applies to all data collections. Written documentation is required to rate the 
following elements. 

a. Data collection procedures and data sources are documented. 

1. Written documentation exists describing the purpose for collecting the 
data. [0/1/2] 

You must review documentation that specifies how each set of data relates 
to one or more of the state ’s indicators. For example, data collected on 
race/ethnicity of students will allow the state to address indicators related 
to disproportionality in LEAs/EIS programs. The rating is “2” if the 
purpose is documented for ALL data collected; the rating is “1 ” if the 
purpose is documented for only SOME data collected; the rating is “0 ” if 
the purpose is NOT documented for any data collected. 

2. Written documentation exists describing the data collection 
methodologies, including site visit plans (if indicator data collection 
involves site visits to LEAs/EIS programs). [0/1/2] 

You must review documentation that specifies how each set of data are 
collected, including descriptions of the data to be collected and the 
procedures for collecting the data (including site visit plans, if 
applicable). For example, how IEPs/IFSPs are reviewed during site visits 
to determine if proper timelines are met, or how parents are surveyed to 
determine level of involvement. Documentation may include data 
collection instruments. The rating is “2 ” if data collection methodologies 
are documented for ALL data collected; the rating is “1 ” if data collection 
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methodologies are documen ted for only SOME data collected; the rating 
is “0” if NO data collection methodologies are documented. 

3. Written documentation exists describing the sources from which data will 
be collected. [0/1/2] 

You must review documentation that specifies where the data used to 
measure performance on indicators originate (e.g., child test data, 
IEPs/IFSPs, parent surveys). For example, race/ethnicity data may come 
from student enrollment forms. The rating is “ 2 ” if sources are 
documented for ALL data collected; the rating is “1 ” if sources are 
documented for only SOME data collected; the rating is “0” if data 
sources are NOT documented for any data collected. 

b. Written documentation exists describing the sampling strategy used to collect 
indicator data. [0/1/2/NA] 

If the indicator data collection involves sampling, you must review 
documentation that explains how a sample is selected to be representative of 
the population. For example, if a sample of IEPs/IFSPs is reviewed, there 
must be a description of the random sampling used to select records. The 
rating is “2 ” if strategies are documented for ALL data collections involving 
sampling; the rating is “1 ” if strategies are documented for only SOME data 
collections involving sampling; the rating is “0” if NO strategies are 
documented for any data collections involving sampling. 

This element should be rated “NA ” if the state does not use sampling for the 
indicator data collection. 

B4. Accurate Data 

Data collection instruments and procedures are reliable and valid for the purposes 
intended. The data collected are examined to ensure that errors are corrected or 
nonexistent. 

General Guidelines: When interviewing the monitoring director or data collection 
manager , ask for a description of the steps taken to ensure that the data are of 
high quality (e.g., piloting conducted, data checked for errors). Review any 
documentation, such as procedural manuals, memos, or reports. Written 
documentation is not required to rate the following elements, except where noted. 

a. Procedures for coding and scoring data are documented. [0/1/2] 

You must review written documentation that explains how data codes are 
assigned and how data are scored. For example, if the state uses dates from 
IEPs/IFSPs for calculating timelines, there must be a description of how to 
calculate the number of days in the time lapse and how to code whether or not 
the target was met. The rating is “2” if documentation of the coding and 
scoring procedures exists for ALL datasets; the rating is “1 ” if documentation 
of the coding and scoring procedures exists for only SOME datasets; the 
rating is “0” if NO documentation of coding and scoring procedures exist. 
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b. Data collection procedures were pilot tested to identify and control sources of 
error. [0/1/2] 

The monitoring director or data collection manager must be able to describe 
how existing indicator data collections were piloted in the field before being 
put into practice, including data collections that have been in place for 
multiple years. For example, piloting could involve sending several districts 
or programs new data items and instructions for review; or, it could involve 
more formalized processes. The rating is “2 ” if data collection procedures 
were piloted for ALL data collections; the rating is “1 ” if data collection 
procedures were piloted for only SOME data collections; the rating is “0 ” if 
NO data collection procedures were piloted. 

c. Procedures are in place to monitor the quality of the data collected. [0/1/2] 

The monitoring director or data collection manager must be able to describe 
a process for reviewing and verifying the accuracy of the data collected. Some 
examples are checking for data entry errors, examining year-to-year changes, 
and assessing coding reliability. The rating is “2 ” if procedures are in place 
to monitoring the quality of the data collected for ALL data collection efforts; 
the rating is “1 ” if procedures are in place to monitor the quality of the data 
collected for only SOME collection efforts; the rating is “0 ” if NO procedures 
are in place to monitor the quality of the data in any data collection effort. 

d. Data provided directly to the state by LEAs/EIS programs are verified for 
accuracy. [0/1/2/NA] 

The monitoring director or data collection manager must be able to describe 
a process for reviewing and verifying the accuracy of the data collected at 
local levels. For example, checking for invalid data, out-of-range codes, or 
missing data. The rating is “2 ” if local data are verified for accuracy for ALL 
data collection efforts; the rating is “1 ” if local data are verified for accuracy 
in only SOME data collection efforts; the rating is “0 ” if local data are NOT 
verified for accuracy in any data collection effort. 

This element should be rated “NA ” if the no indictor data are provided 
directly to the state by LEAs/EIS programs. 

B5. Analysis of Indicator Data 

Indicator data are analyzed to detennine LEA/EIS program performance. 

General Guidelines: This key characteristic refers to quantitative data only. 
Quantitative data are in numerical form (e.g., percentages, child counts, number 
of days) and qualitative data are not (e.g., text, photographs, videos). However, 
qualitative data can be converted to quantitative form (e.g., when reviewing 
IEPs/IFSPs for signatures, yes=l, no=0). Note that for this rating exercise, our 
interview data (qualitative data) are being converted to quantitative form. 
Quantitative data can be either compliance data or performance/outcome data. 
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When interviewing the monitoring director or data collection manager, ask for a 
description of the steps taken to analyze the indicator data. Review any 
documentation, such as internal reports or memos. Written documentation is not 
required to rate the following elements. 

a. Preliminary analyses are conducted to gain a thorough understanding of the 
indicator data. [0/1/2] 

The monitoring director or data collection manager must be able to describe 
analyses conducted on the indicator data to examine data variability within or 
across indicators. For example, means, medians, frequencies, and ranges are 
examined, and outliers are identified. The rating is “2” if analyses are 
conducted for ALL indicators; the rating is “1 ” if analyses are conducted for 
only SOME indicators; the rating is “0” if NO analyses are conducted. 

b. Analyses are conducted to assess performance on the indie ators/targets. 

[0/1/2] 

The monitoring director or data collection manager must be able to describe 
the analyses used to examine statewide or LEA/EIS program performance on 
each indicator. These analyses may also include examining patterns within or 
across LEAs/EIS programs. For example, the state may calculate percentages 
or present frequencies by indicator for each LEA/EIS program. The rating is 
“ 2 ” if such analyses are conducted for ALL indicators; the rating is “1 ” if 
such analyses are conducted for only SOME indicators; the rating is “0” if 
NO such analyses are conducted. 

B6. Practical and Efficient Procedures 

Indicator data collection procedures are practical and minimize burden on the 
LEA/EIS program, including limiting disruption of daily routines of schools or 
programs. 

General Guidelines: When interviewing the monitoring director or data collection 
manager, ask about how the data collected are used in relation to measuring 
performance on indicators, how the data collection efforts are respectful of school 
and program schedules, and what steps are taken to minimize paperwork 
necessary on the part of LEAs/EIS programs. Written documentation is not 
required to rate the following elements. 

a. No extraneous data are collected (i.e., all data collected are used to measure 
performance on the selected indicators). [0/1/2] 

The monitoring director or data collection manager must describe how the 
data collection efforts are limited to just those data that are pertinent to 
reporting on the indicators. That is, time was not wasted collecting extraneous 
information or data. Extraneous refers to information that was irrelevant and 
involved a substantive data collection effort. The rating is “2 ” if it appears 
that ALL data collection efforts are limited to data that is used in for indicator 
data collection and analysis; the rating is “1 ” if SOME efforts appear to 
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collect extraneous data; the rating is “0” if ALL efforts appear to collect 
extraneous data. 

b. Procedures are in place to minimize interruptions/disruptions to school, 
program, or district routines when collecting data. [0/1/2] 

The monitoring director or data collection manager must describe how the 
data collection procedures were designed to minimize disruptions to local 
schools/programs. For example, when scheduling site visits, LEAs/EIS 
programs are given options for date/times, or checklists/forms are developed 
to expedite local data collection. The assessment of minimal disruption to 
local school/programs is subjective. Therefore, it is important to recognize 
that the judgment involves several factors, including (1) the importance of the 
data, (2) the efficiency of the method of obtaining the data, and (3) the 
reliability of the data collection method (e.g., teacher ratings versus student 
direct assessment). The rating is “2” if efforts to minimize disruptions are 
apparent for ALL data collection efforts; the rating is “1 ” if efforts to 
minimize disruptions are apparent for only SOME data collection efforts; the 
rating is “0 ” if it appears that disruptions are NOT a consideration in any 
data collection efforts. 

C. Problem Detection 

Cl, Defensible Findings 

The findings drawn from the problem identification process are adequately 
defended and supported by comparing LEA/EIS program performance to 
specified targets. 

General Guidelines: Written documentation is required to assess the extent to 
which problem identification findings are defensible. Written documentation can 
take the form of a report, memo, letter, PowerPoint presentation, or similar 
documentation. If no written documentation exists, then the rating is a “0. ” 

a. Findings reflect perfonnance in relation to specific targets. [0/1/2] 

You must review documentation that confirms that the data collected and the 
target are comparable and that they are actually compared. For example, if 
the indicator is the percentage of students who pass the eighth grade 
statewide reading assessment and the target is 84% (with an implied range of 
84%-100%) and the finding is 83% of students taking the eighth grade 
reading assessmen t in District X met the standard for a passing grade, this 
would be a fair and accurate assessment. On the other hand, if the state has 
this same indicator and target, but the finding is that in district X, student 
performance fell in the bottom 2%> of all districts in the state, this would not 
be a fair and accurate assessment because the findings ignore the target. The 
rating is “ 2 ” if findings reflect a fair and accurate comparison between the 
performance and the target for ALL indicators; the rating is “1 ” if findings 
reflect a fair and accurate comparison between the performance and target 
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for only SOME indictors; the rating is “ 0 ” if NONE of the findings reflect a 
fair and accurate comparison between the performance and the target. 

b. Findings are limited to specific time periods. [0/1/2] 

You must review documentation to confirm that all findings refer to data that 
were collected during the same time frames (i.e., for the 2004-2005 
monitoring cycle). The rating is “ 2 ” if findings are limited to specific time 
period for ALL indicators; the rating is “1 ” if findings are limited to specific 
time periods for SOME indicators; the rating is “0” if findings are NOT 
limited to specific time periods for any indicators. 

c. Findings are adequately supported by data. [0/1/2] 

You must review documentation that confirms the discussion of findings is 
supported by the data. The discussion is supported if it accurately describes 
the results. The reviewer may need to carefully read the discussion to 
determine whether or not it makes sense given the data that are presented. If 
no data are presented then the reviewer must assume the discussion is NOT 
adequately supported. The rating is “2” if discussions of ALL indicators 
present an accurate reflection of the data; the rating is “1 ” if discussions of 
SOME indicators present an accurate reflection of the data; the rating is “0 ” 
if NO discussions of indicators present an accurate reflection of the data. 

C2. Complete Reporting on the State 

A report completely describes the problems that are identified throughout the state 
regarding perfonnance on specified indicators and targets, so that essential 
information can be provided and is easily accessible to stakeholders. 

General Guidelines: Do not confuse Defensible Findings with Complete 
Reporting. Defensible Findings concerns a specific quality of the discussion about 
findings related to problem identification. Complete Reporting concerns the 
existence of a written documen t with all the importan t elements needed to convey 
a complete understanding of identified problems associated with state indicators 
and targets. 

Review documentation to assess the extent to which the state-level report on 
identified problems is complete. The report should present findings as they relate 
to the state overall (i.e., a summation of identified problems across LEAs/EIS 
programs so the “health ” of the state can be understood). Written documen tation 
is required to rate the following elements, except where noted. Written 
documentation can take the form of a report, memo, letter, PowerPoint 
presentation, or similar documentation. Any of these forms are referred to as the 
“report. ” If the state does not have a state-level report, then you should rate all 
the elements in this section as “NO. ” 

a. State reports focus on the specified indicators and targets (i.e., discusses 
performance in relation to targets). [YES/NO] 
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For this rating to be a “YES, ” the report must discuss performance as it 
relates to the state ’s indicators and targets. For example, the report should 
include a discussion of how many LEAs/EIS programs fell below the target for 
a specified indicator and how many performed at a level at or above the 
target. 

b. State reports describe the data used to assess performance on specified 
indicators and targets. [YES/NO]. 

For this rating to be a “YES, ” the report must include a discussion of what 
data are used to arrive at a measure of performance. At a minimum, the 
report presents a discussion of the nature (survey, observation, etc.) and 
original source of the data (student, teacher, etc.). 

c. State reports describe the methods and procedures used to identify problems. 
[YES/NO] 

For this rating to be a “YES, ” the report must include a discussion of the 
methods and procedures used to arrive at a comparison of the performance(s) 
to the target(s). At a minimum, the report should describe the general data 
collection strategy and the process by which data are aggregated and 
compared to the target. 

d. State reports describe limitations of the data used to identify problems. 
[YES/NO] 

For this rating to be a “YES, ’’ the report must include a discussion of the 
limitations of the data. For example, the report may discuss the accuracy, 
validity, and reliability of the data collection instruments. If sampling is used, 
then there may be a discussion of how representative the sample is and 
whether the findings can be generalized. 

e. Visual displays in state reports are clear and understandable and relate to the 
findings/data. [YES/NO/NA] 

For this rating to be a “YES, ” visual displays should be straightforward 
(accurately labeled) and offer the reader a clear understanding of the 
findings/data. 

A rating of “NA ” means there were no visual displays of data in the report. 

f. State reports describe the extent of the problem in the state. [YES/NO] 

For this rating to be a “YES, ” the report must address the “health ” of the 
state with regard to all the indicators. That is, the report must include a 
discussion of the findings as they relate to the context of the entire state. 

g. Reports include a discussion of strengths and areas needing improvement. 
[YES/NO] 

For this rating to be a “YES, ” the report must address the strengths and areas 
needing improvement with regard to all the indicators. 



C-15 




h. Someone who did not write the report or collect the data reviews the report. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must confirm that an 
independent reviewer was used in the report preparation process before the 
report was disseminated to an intended audience. That is, someone other than 
the report’s author or participant in the data collection process read the 
report and provided feedback on its clarity and completeness. You do not need 
to see written documentation to rate this element. 

C3. Complete Reporting on LEA/EIS Programs 

Reports prepared by the state completely describe the problems that are identified 
regarding LEA/EIS program performance on specified indicators and targets, so 
that essential information can be provided and is easily accessible to local 
stakeholders. 

General Guidelines: Do not confuse Defensible Findings with Complete 
Reporting. Defensible Findings concerns a specific quality of the discussion about 
findings related to problem identification. Complete Reporting concerns the 
existence of a written documen t with all the importan t elemen ts needed to convey 
a complete understanding of identified problems associated with state indicators 
and targets. 

Review documentation to assess the extent to which reports prepared by the state 
on problems identified at the local level are complete. These reports should 
present findings regarding problems identified within individual LEAs/EIS 
programs (e.g., district/program report cards). Written documentation is required 
to rate the following elements, except where noted. Written documentation can 
take the form of a report, memo, letter, PowerPoint presentation, or similar 
documentation. Any of these forms are referred to as the “report.” If the state 
does not have local-level reports, then you should rate all the elements in this 
section as “NO. ” 

The dichotomous scale (YES/NO) is applied to the elements under this key 
characteristic because it is assumed that the state uses a common template or 
model for all local reports. 

a. Local-level reports focus on the specified indicators and targets (i.e., discusses 
performance in relation to targets). [YES/NO] 

For this rating to be “YES, ” the report must discuss LEA/EIS program 
performance on the indicators and targets, and include a presentation of 
findings associated with the performance of LEAs/EIS programs on the 
indicators. Note that each finding should be associated with an 
indicator/target. 

b. Local-level reports describe the data used to assess perfonnance on specified 
indicators and targets. [YES/NO] 
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For this rating to be “YES, ” the report must include a discussion of what data 
are used to arrive at a measure of performance. At a minimum, the report 
presents a discussion of the nature (survey, observation, etc.) and original 
source of the data (student, teacher, etc.). 

c. Local-level reports describe the methods and procedures used to identify 
problems. [YES/NO] 

For this rating to be “YES, ” the report must include a discussion of the 
methods and procedures used to arrive at a comparison of the performance (s) 
to the target(s). At a minimum, the report should describe the general data 
collection strategy and the process by which data are aggregated and 
compared to the target. 

d. Local-level reports describe limitations of the data used to identify problems. 
[YES/NO] 

For this rating to be “YES, ” the report must include a discussion of the 
limitations of the data. For example, the report may discuss the accuracy, 
validity, and reliability of the data collection instruments. If sampling is used, 
then there may be a discussion of how representative the sample is and 
whether the findings can be generalized. 

e. Visual displays in local-level reports are clear and understandable and relate 
to the findings/data. [YES/NO/NA] 

For this rating to be a “YES, ” visual displays should be straightforward 
(accurately labeled) and offer the reader a clear understanding of the 
findings/data. 

A rating of “NA ” means there were no visual displays of data in the report. 

f. Reports on LEAs/EIS programs include a discussion of strengths and areas 
needing improvement. [YES/NO] 

For this rating to be a “YES, ” the report must address the strengths and areas 
needing improvement with regard to all the indicators. 

g. Someone who did not write the report or collect the data reviews the report. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must confirm that an 
independent reviewer was used in the report preparation process before the 
report was disseminated to an intended audience. That is, someone other than 
the report’s author or participant in the data collection process read the 
report and provided feedback on its clarity and completeness. You do not need 
to see written documentation to rate this elements. 

C4. Dissemination of Problem Identification Reports to Stakeholders 

Problem identification reports are accessible and available to stakeholders. 

Interview the monitoring director about the availability of problem identification 
reports to stakeholders, and whether discussions occur between stakeholders and 
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SEA/lead agency staff. Further explore accessibility and availability by 
confirming what reported dissemination means (such as examining websites) and 
through interviews with stakeholders and LEA/EIS program staff 

a. State-level problem identification reports are disseminated to stakeholders. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe actions 
taken to disseminate state-level problem identification reports to stakeholders, 
including the state-level stakeholder committee. Note: If problem 

identification reports as available upon request, this is not a formal 
dissemination process and the rating would be “NO. ” 

b. Local level problem identification reports are disseminated to local 
stakeholders beyond the LEA/EIS program administrators. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe actions 
taken to formally disseminate local-level problem identification reports to 
local stakeholder groups (local coordinating councils or advisory panels, 
working groups, etc.) stakeholders. Note: If problem identification reports as 
available upon request, this is not a formal dissemination process and the 
rating would be “NO." 

c. Opportunities are provided for stakeholders to discuss state-level problem 
identification reports with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe 
opportunities for stakeholders, including the state-level stakeholder 
committee, to discuss problem identification reports with SEA/lead agency 
staff. These opportunities need not be formal opportunities; that is, they may 
include e-mails or telephone conversations with SEA/lead agency staff. 

d. Opportunities are provided for LEA/EIS program personnel to discuss local- 
level problem identification reports with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ’’ the monitoring director must describe 
opportunities for LEA/EIS program personnel to discuss local level problem 
identification reports with SEA/lead agency staff. These opportunities need 
not be formal opportunities ; that is, they may include e-mails or telephone 
conversations with SEA/lead agency staff. 

e. Opportunities are provided for local stakeholders to discuss local-level 
problem identification reports with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ’’ the monitoring director must describe 
opportunities for local stakeholder groups to discuss local level problem 
identification reports with state agency staff. These opportunities need not be 
formal opportunities; that is, they may include e-mails or telephone 
conversations with SEA/lead agency staff. 
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Component: Problem Investigation 

D. Problem Investigation 

Dl. Stakeholder Participation 

Stakeholders are informed and contribute throughout the problem investigation 
process. 

General Guidelines: Stakeholders should contribute to the problem investigation 
process. The state should have a process for seeking input from a variety of 
stakeholders, depending on the nature or prevalence of the problem under 
investigation. When interviewing the monitoring director and gathering 
information on the problem investigation process, assess stakeholder involvement 
in this process (e.g., provide guidance on what additional data should be 
collected, participate in site visits, review investigation findings, etc.) Review any 
written documentation of stakeholder involvement (i.e., memos, meeting minutes, 
copies of materials/reports distributed to members), but written documentation is 
not required to rate the following elements. Interview a parent or advocate who 
participated in the problem investigation process before finalizing your ratings. 

a. Representatives from key constituencies (including advocacy groups, parents 
of children with disabilities, teachers, service providers, LEA/EIS program 
and school administrators) participate in the process. 

1 . Input is sought from parents of children with disabilities or representatives 
from advocacy groups. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for obtaining input from parents of children with disabilities or 
disability advocates (e.g., a survey, sending out materials for feedback, 
meetings). These parents/advocates should represent different disability 
categories. 

2. Parents of children with disabilities or the representatives from advocacy 
groups represent the diversity that exists in the state. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe how 
state diversity is represented among the parents of children with 
disabilities and disability advocate who provide input into the problem 
investigation process. This will vary depending on the demographics of 
the state. Some examples are representatives from rural and urban areas 
or representatives from different racial/ethnic groups. 

3. Materials used to seek input from stakeholders are tailored to improve 
understanding and facilitate their contribution to the problem investigation 
process. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for providing additional background information and explaining 
technical terms/jargon to stakeholders. Such a process might include a 



C-19 




review of materials by someone who is expected to think about potential 
problems with readability or accessibility of the information. If the 
monitoring director indicates that this is not necessary or that nothing 
special is done to tailor materials, then the rating would be a “NO. ” 

4. Input is sought from direct service personnel (i.e., teachers, service 
providers, and principals). [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for obtaining input from direct sendee personnel (e.g., a survey, 
sending out materials for feedback, meetings). 

b. LEA/EIS program administrators (e.g., superintendents, special education/ 
early intervention supervisors) contribute to the problem investigation 
process. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe how 
LEA/EIS program administrators are involved in the problem investigation 
process. 

c. A state leadership figure from special education or early intervention (e.g., 
state director of special education, Part C lead agency coordinator, the state 
monitoring director) are informed (are aware) of the problem investigation 
process. [YES/NO] 

For this rating to be a “YES, ’’ the monitoring director must indicate that, at 
minimum, a state leadership figure is kept informed of the problem 
investigation process (e.g., through reports, conference calls, meetings) if 
problem investigation is a state-driven process being conducted at the local 
level. If the state is conducting problem investigations, then the rating is 
“YES. ” 

D2. Credibility of Problem Investigation Team 

The individuals responsible for investigating identified problems possess the 
requisite technical competence, substantive knowledge, and experience. 

General Guidelines: When interviewing the monitoring director, determine who is 
responsible for managing and conducting problem investigations. Ask about the 
experience and training of these individuals. Review any documentation on 
training of those participating in the problem identification process, but written 
documentation is not required to rate these elements. If the monitoring director 
does not manage the data collection, you should also interview the problem 
investigation manager(s). 

a. Managers (of problem investigations) have a background and knowledge 
related to identified problems. [YES/NO] 

For the rating to be a “YES, ” the monitoring director or problem 
investigation manager must confirm that the persons leading problem 
investigations either have the training or experience to lead the investigative 
process or have the support of persons who do. 
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b. Individuals conducting problem investigations have appropriate training. 
[YES/NO] 

For the rating to be a “YES, ” the monitoring director or problem 
investigation manager must confirm that persons participating in problem 
investigations have at least one opportunity to receive training related to the 
investigation at hand and that they participated in training. Training in this 
context refers to any class or orientation to the investigative process 
developed by the state office or lead agency or to the general process of 
conducting research activities. 

D3. Practical and Efficient Procedures 

Data collection procedures for problem investigations are practical, internally 
consistent, and efficient while minimizing burdens on the LEA/EIS program, 
including the disruption of daily routines of schools or programs. 

General Guidelines: When interviewing the monitoring director or problem 
investigation manager, ask about how the data collected are used in relation to 
problem investigations, how the problem investigation data collection efforts are 
respectful of school and program schedules, and what steps are taken to minimize 
paperwork necessary on the part of LEAs/EIS programs. Review any relevant 
documentation, but written documentation is not required to rate the following 
elements. 

a. No extraneous data are collected (i.e., all data collected are used to investigate 
identified problems). [YES/NO] 

For the rating to be a “YES, ” the monitoring director or problem 
investigation manager must describe how the data collection efforts are 
limited to just those data intended for use in the problem investigation, unless 
an otherwise documented need was noted. That is, time was not wasted 
collecting extraneous information or data. Extraneous refers to information 
that was irrelevant to the investigation and involved a substantive data 
collection effort. Extraneous data collection does not include relevant data 
collection, such as potential explanatory variables or variables used to track 
subject identification. 

b. Procedures are in place to minimize interruptions/disruptions to school, 
program, or district routines when collecting data. [YES/NO] 

For the rating to be a “YES, ” the monitoring director or problem 
investigation manager must describe how the data collection procedures were 
designed to minimize disruptions to local schools/programs. For example, 
when scheduling site visits, LEAs/EIS programs are given options for 
dates/times, or checklists /forms are developed to expedite local data 
collection. The assessment of minimal disruption to local school/programs is 
subjective. Therefore, it is important to recognize that the judgment involves 
several factors, including (1) the importance of the data, (2) the efficiency of 
the method of obtaining the data, and (3) the reliability of the data collection 
method (e.g., teacher ratings versus student direct assessment). 
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D4. Thoroughly Documented Problem Investigations 

The process used to investigate identified problems is described in sufficient 
detail so that the utility, reliability, and validity of the information can be 
assessed. 

General Guidelines: Review documentation (i.e., user manuals, reports, memos) 
related to the problem investigation process. Clarify that documentation applies 
to all data collections. Written documentation is required to rate the following 
elements. 

a. Each problem being investigated is clearly defined and documented. 
[YES/NO] 

For this rating to be a “YES, ’’you must review documentation to confirm that 
the problem(s) being investigated were plainly elaborated in sufficient detail 
to permit an understanding of what the problem is (was), and how and why it 
was identified. 

b. Data collection methodologies and data sources are documented. 

1. Written documentation exists describing the purpose for collecting the 
data. [YES/NO] 

For the rating to be a “YES, ” you must review documentation that 
specifies how each set of data relates to one or more of the problems being 
investigated. 

2. Written documentation describes data collection methodologies including 
site visit plans (if problem investigations involve site visits to LEAs/EIS 
programs). [YES/NO] 

For this rating to be a “YES, ” data collection methodologies must be 
documented. You must review documentation that specifies how each set 
of data are collected, including descriptions of the data to be collected 
and the procedures for collecting the data (including site visit plans, if 
applicable). For example, how IEPs/IFSPs are reviewed during site visits 
to determine if proper timelines are met, or how parents are surveyed to 
determine level of involvement. Documentation may include data 
collection instruments. 

c. Written documentation exists describing the sources from which data will be 
collected. [YES/NO] 

For this rating to be a “YES, ” the sources of data are documented for all data 
collected. You must review documentation that specifies where the data 
collected for problem investigations originate (e.g., child test data, 
IEPs/IFSPs, parent surveys). 

d. Written documentation exists describing the sampling strategy used to 
investigate identified problems. [YES/NO/NA] 
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For this rating to be a “YES, ” you must review documentation that provides 
guidelines for the use of sampling in problem investigations. The 
documentation must explain how a sample is selected to be representative of 
the population. For example, if a sample of IEPs/IFSPs is reviewed for 
problem identification, there must be a description of the sampling used to 
select records. 

This element should be rated “NA ” if problem investigations do not involve 
sampling. 

D5. Accurate Data 

Data collection instruments and procedures are reliable and valid for the purposes 
intended. The data collected are examined to ensure that errors are corrected or 
nonexistent. 

General Guidelines: When interviewing the monitoring director or problem 
investigation manager, ask about procedures that are in place to ensure any data 
collected to investigate identified problems are accurate. Such procedures may 
include systematic coding and scoring procedures, monitoring the quality of data, 
and verifying/checking data submitted by other parties (e.g., data provided by 
LEAs/EIS programs). Written documentation is not required to rate the following 
elements, except where noted 

a. Procedures for coding and scoring data are documented. [YES/NO] 

For this rating to be a “YES, ” documentation of coding and scoring 
procedures exists. You must review written documentation that explains how 
data codes are assigned and how data are scored. For example, if using dates 
from IEPs/IFSPs for calculating timelines, there must be a description of how 
to calculate the number of days in the time lapse and how to code whether or 
not the target was met. 

b. Procedures are in place to monitor the quality of the data collected. [YES/NO] 

For this rating to be a “YES, ” procedures are in place to monitor the quality 
of the data collected for all data collection efforts. The monitoring director or 
problem investigation manager must be able to describe a process for 
reviewing and verifying the accuracy of the data collected. Some examples are 
checking for data entry errors, examining year-to-year changes, and 
assessing coding reliability. 

c. Data are verified for accuracy. [YES/NO] 

For the rating to be a “YES, ’’ collected data are verified for accuracy. The 
monitoring director or problem investigation manager must be able to 
describe a process for reviewing and verifying the accuracy of the data 
collected at local levels. For example, checking for invalid data, out-of-range 
codes, or missing data. 
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D6. Analysis of Quantitative Data 

The analytic techniques fit the quantitative data and the identified problems under 
investigation. 

General Guidelines: Interview the monitoring director or problem investigation 
manager to confirm that any quantitative data collected during the problem 
investigation process are analyzed and that the analyses are designed to gain 
more understanding about identified problems. The intent of this key 
characteristic is to ensure that quantitative data are analyzed (not just collected) 
to gain further understanding of identified problems. Written documentation is 
not required to rate the following elements. 

Quantitative data are in numerical form (e.g., percentages, child counts, number 
of days) and qualitative data are not (e.g., text, photographs, videos). However, 
qualitative data can be converted to quantitative form (e.g., when reviewing 
IEPs/IFSPs for signatures, yes=l, no=0). Note that for this rating exercise, our 
interview data (qualitative data) are being converted to quantitative form. 
Quantitative data can be either compliance data or performance/outcome data. 

a. Exploratory analyses are conducted to ensure the data’s correctness and to 
gain an increased understanding of the data. [YES/NO/NA] 

For this rating to be a “YES, ” the monitoring director or problem 
investigation manager must confirm that basic analyses of quantitative data 
that examine outliers and anomalies are conducted to ensure the accuracy of 
the data (i.e., basic descriptive statistics are used). 

The rating of “NA ” means that no quantitative data are collected for problem 
investigations. 

b. Data variability is examined. [YES/NO/NA] 

For this rating to be a “YES, ” the monitoring director or problem 

investigation manager must confirm that analyses are conducted to examine 
data variability, such as standard deviations or variance. 

The rating of “NA ” means that no quantitative data are collected for problem 
investigations. 

c. Statistics presented in reports or presentations reflect the nature of the data 
and the identified problems under investigation. [YES/NO/NA] 

For this rating to be a “YES, ” the monitoring director or problem 

investigation manager must describe a process or steps that are taken to 
ensure that any statistics included in reports or presentations on problem 
investigations are clearly presented and directly related to the data. 

The rating of “NA ” means that no quantitative data are collected for problem 
investigations. 



C-24 




D7. Analysis of Qualitative Data 

The analytic techniques fit the qualitative data and the identified problems under 
investigation. 

General Guidelines: You should interview the monitoring director or problem 
investigation manager to confirm that any qualitative data collected during the 
problem investigation process are analyzed and that the analyses are designed to 
gain more understanding about identified problems. The intent of this key 
characteristic is to ensure that qualitative data are analyzed (not just collected) to 
gain further understanding of identified problems. Written documentation is not 
required to rate the following elements. 

a. Multiple methodologies are used to verify the accuracy of findings (e.g., 
triangulation). [YES/NO/NA] 

The monitoring director or problem investigation manager must describe how 
findings or themes emerging from analyses of qualitative data are verified 
using more than one data source or data collection method. 

The rating of “NA ” means that no qualitative data were collected for problem 
investigations. 

b. Analytic procedures and methods of summarization are used. [YES/NO/NA] 

The monitoring director or problem investigation manager must describe the 
procedures used to analyze and summarize qualitative data. For example, if 
the problem is a low percentage of children receiving services in natural 
environments, service providers may be interviewed regarding challenges 
they face. These interview data should be summarized so that is it clear what 
the main challenges are. 

The rating of “NA ” means that no qualitative data were collected for problem 
investigations. 

D8. Defensible Findings 

The findings from the problem investigation process are adequately documented, 
defended, and supported by data, so that the state can proceed with improvement 
planning or corrective actions. 

General Guidelines: Review documentation to assess the extent to which findings 
resulting from the problem investigation process are defensible. Documentation 
can take the form of a report, memo, letter, PowerPoint presentation, or similar 
documentation. Written documentation is required to rate the following elements. 

a. Findings focus directly on the identified problems under investigation. 
[YES/NO] 

For this rating to be a “YES, ” the findings resulting from problem 
investigations shoidd be discussed in relation to the original identified 
problems. Remember the purpose of a problem investigation is to find out why 
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a problem exists. So the findings should reflect why an identified problem 
exists. 

b. Findings and conclusions are adequately supported by data. [YES/NO] 

For this rating to be a “YES, ’’findings from the problem investigation must 
be supported bv specific data. If no data are presented, then the rating is 
“NO. ” 

Component: Corrective Action and Enforcement 

E. Corrective Action and Enforcement 

Note: Only information in Corrective Action Plans (CAPs) related to specific instances of 
noncompliance should be used in rating these elements. Any information in CAPs related 
to improvement planning resulting from noncompliance should be rated under 
Improvement Planning, not Corrective Action and Enforcement. 

The dichotomous scale (YES/NO) is applied to the elements under Corrective Action and 
Enforcement because the state should have a common model, template, or guidelines for 
all CAPs. 

El. Detailed Corrective Action Plans 

Corrective action plans (CAPs) address the identified problem(s) related to 
procedural noncompliance and are sufficiently detailed. 

General Guidelines: Review the template for corrective action plans or review 
examples of corrective action plans that were developed for specific LEAs/EIS 
programs. Interview the monitoring director to confirm your observations on how 
the state uses corrective action plans to address procedural noncompliance. 
Written documentation is required to rate the following elements. 

a. LEA/EIS program CAPs address the identified problem(s) related to 
procedural noncompliance. 

1. CAPs specify all problem areas. [YES/NO] 

For this rating is to be a “YES, ” all areas or instances of noncompliance 
identified are listed in the CAP. If you are reviewing a template for a CAP, 
there must be a portion of the template devoted to specifying the areas that 
need to be addressed. 

2. Written action plans address each problem area. [YES/NO] 

For this rating to be a “YES, ” it must be clear which actions are intended 
to remedy which problem areas. For example, if the problem area is no 
evidence of transition planning at age 14, then the CAP should include the 
actions that will be taken to address this problem. 
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b. LEA/EIS program CAPs are detailed and include steps/components and 
timelines. 

1. Steps/components of the CAPs are clearly delineated for each identified 
problem. [YES/NO] 

For this rating to be a “YES, ” specific steps/components are included in 
CAPs for ail identified problems. For example, the district may first be 
required to develop a form to document transition planning, and then 
conduct training on using the form to guide their planning. 

2. Timelines are included for each step/component of the CAPs. [YES/NO] 

For this rating to be a “YES, ” a timeline or date for completion must 
accompany every step/component of the CAP. For example, the form to 
document transition planning must be developed by June 30 th , 2005 and 
the training completed by September I st , 2005. 

3. Plans to measure progress toward meeting timelines are documented. 
[YES/NO] 

For this rating to be a “YES, ’’ there must be specific means (e.g., memos, 
calls with state monitors, etc.) for the LEA/EIS program to document 
progress toward meeting the timeline that is specified in the CAP. Note, 
this does not need to be a formal, written document. Documentation might 
include telephone progress reports or e-mail progress reports. 

E2. Dissemination of Corrective Action Plans to Stakeholders 

Corrective action plans (CAPs) are accessible and available to local stakeholders. 

General Guidelines: Interview the monitoring director to determine how 
stakeholders are made aware of CAPs resulting from noncompliance. Also, 
determine the extent to which these issues are discussed with LEAs/EIS programs 
and other stakeholders. You should further explore accessibility and availability 
by confirming reported dissemination means (such as examining websites) and 
through interviews with LEA/EIS program staff or other stakeholders. Written 
documentation is not required to rate the following elements. 

a. CAPs are disseminated to local stakeholders beyond the LEA/EIS program 
administrators. [YES/NO] 

For this rating to be a “YES, ” the state must require that CAPs be formally 
disseminated to local stakeholder groups (local coordinating councils or 
advisory panels, working groups, etc.). For example, this could mean that the 
state posts all CAPs on its website or that the state specifies a process that the 
LEA/EIS program must use to disseminate CAPs. Note: If CAPs are available 
upon request, this is NOT a formal dissemination process and the rating 
would be “NO. ” 
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b. Opportunities are provided for LEA/EIS program personnel to discuss CAPs 
with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must indicate that 
LEA/EIS program personnel are given opportunities to discuss the results 
with state agency staff. These opportunities need not be formal opportunities ; 
that is, they may include e-mails or telephone conversations with SEA/lead 
agency staff or exit interviews or conversations at the end of site visits. 

c. Opportunities are provided to local stakeholders to discuss CAPs with 
SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must indicate that local 
stakeholder groups are given opportunities to discuss the results with state 
agency staff. These opportunities need not be formal opportunities; that is, 
they may include e-mails or telephone conversations with SEA/lead agency 
staff. 

E3. Enforcement of Corrective Action Plans 

Enforcement procedures are in place to ensure compliance with needed corrective 
action. 

General Guidelines: Review documentation that discusses the enforcement 
procedures available to the state to ensure areas of noncompliance are 
addressed. Written documentation is required to rate the following elements. 

a. Enforcement procedures are documented. 

1. The SEA/lead agency has written general enforcement procedures. 
[YES/NO] 

For this rating to be a “YES, ’’ the documentation must discuss general 
enforcement procedures. The general enforcement procedures must 
include statements noting that the state agency will assess whether 
corrections have been made and that failure to do so will result in 
sanctions. 

2. General enforcement procedures delineate specific actions that will be 
taken by the SEA/lead agency if a CAP is not implemented. [YES/NO] 

For this rating to be a “YES, ” the documentation must confirm that the 
general enforcement procedures include sanctions (specific actions to be 
taken if a CAP is not implemented) . Sanctions include actions such as 
withholding funds, SEA assumption of the provision of special education 
services in a district, denial of future early intervention sendee licenses, 
etc. If there are no written general enforcement procedures, you should 
rate this element as “NO. ” 
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b. General enforcement procedures are accessible to stakeholders. [YES/NO] 

For this rating to be a “YES, ” the general enforcement procedures must be 
written in user-friendly terms (i.e., not legalese). For example, “accessible’’ 
enforcement procedures cannot just quote directly from state law or 
regulations. 



Component: Improvement Planning and Implementation 

F. Improvement Planning 

Note: Any information in CAPs related improvement planning resulting from 
noncompliance should be rated under Improvement Planning, not Corrective Action and 
Enforcement. 

The dichotomous scale (YES/NO) is applied to the elements under Improvement Planning 
because it is assumed that the state has a structure in place to guide the development of 
improvement plans at the local level. Remember, you are making ratings based on that 
structure and not each individual LEA/EIS program improvement plan. Also, your 
ratings should not pertain to instances of state-wide improvement planning. 

FI. Stakeholder Participation 

Stakeholders are informed and contribute throughout the improvement planning 
process. 

General Guidelines: Stakeholders should contribute to the improvement planning 
process. The state should have a process for seeking input from a variety of 
stakeholders, depending on the nature of the improvement planning. When 
interviewing the monitoring director and gathering information on the 
improvement planning process, assess stakeholder involvement in this process 
(e.g., provide guidance on improvement strategies, steps in the plan, timelines). 
Review any written documentation of stakeholder involvement (i.e., memos, 
meeting minutes, copies of materials/reports distributed to members), but written 
documentation is not required. Interview a parent or advocate who participated 
in the improvement planning process before finalizing your ratings. 

a. Representatives from key constituencies (including advocacy groups, parents 
of children with disabilities, teachers, service providers, LEA/EIS program 
and school administrators) participate in the process. 

1 . Input is sought from parents of children with disabilities or representatives 
from advocacy groups. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for obtaining input from parents of children with disabilities or 
disability advocates (e.g., a survey, sending out materials for feedback, 
meetings). These parents/advocates should represent different disability 
categories. 



C-29 




2. Parents of children with disabilities or the representatives from advocacy 
groups represent the diversity that exists in the state. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe how 
state diversity is represented among the parents of children with 
disabilities and disability advocate who provide input into the 
improvement planning process. This will vary depending on the 
demographics of the state. Some examples are representatives from rural 
and urban areas or represen tatives from differen t racial/ethnic groups. 

3. Materials used to seek input from stakeholders are tailored to improve 
understanding and facilitate their contribution to the improvement 
planning process. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for providing additional background information and explaining 
technical terms/jargon to stakeholders. Such a process might include a 
review of materials by someone who is expected to think about potential 
problems with readability or other accessibility of the information. If the 
monitoring director indicates that this is not necessary or that nothing 
special is done to tailor materials, then the rating would be a “NO. ” 

4. Input is sought from direct service personnel (i.e., teachers, service 
providers, and principals). [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe a 
process for obtaining input from direct sendee personnel (e.g., a survey, 
sending out materials for feedback, meetings). 

b. LEA/EIS program administrators (e.g., superintendents, special 
education/early intervention supervisors) contribute to the improvement 
planning process. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe how 
LEA/EIS program administrators are involved in the improvement planning 
process. 

c. A state leadership figure from special education or early intervention (e.g., 
state director of special education, Part C lead agency coordinator, the state 
monitoring director) is informed (is aware) of the improvement planning 
process. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must indicate that, at 
minimum, a state leadership figure is kept informed of the improvement 
planning process (e.g., through reports, conference calls, meetings) if 
improvement planning is a state-driven process being conducted at the local 
level. If the state is conducting improvement planning, then the rating is a 
“YES. ’’ 
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F2. Detailed Improvement Plans 

Improvement plans address the identified problems and are sufficiently detailed. 

General Guidelines: You should review a template for improvement plans or 
review examples of LEA/EIS program improvement plans to determine whether 
the state requires that plans include all the identified problems, details regarding 
the specific steps to be taken, and timelines to be followed. Written documentation 
is required to rate the following elements. 

a. LEA/EIS program improvement plans address identified problems. 

1. Improvement plans specify all problem areas. [YES/NO] 

For this rating to be a “YES, ” improvement plans must specify all of the 
problem areas to be addressed. If you are reviewing a template or a form 
for improvement planning, there must be a section devoted to delineating 
the problem areas. 

2. Written improvement plans address each problem area. [YES/NO] 

For this rating to be a “YES, ” it must be clear which activities are 
intended to improve which identified problems. For example, if the 
problem area is poor performance for the district on state-wide reading 
assessments, then the improvement plan should specify which activities 
will be conducted to address this problem. 

b. Improvement plans include steps/components and timelines. 

1. Steps/components of the improvement plans are clearly delineated for 
each identified problem. [YES/NO] 

For this rating to be a “YES, ” specific steps/components are included in 
improvement plans for all identified problems. For example, workshops 
will be provided for teachers on the reading/language arts curriculum 
being used in the district and the district will hire additional instructional 
assistance for the lowest performing schools. 

2. Timelines are included for each step/component of the improvement plans. 
[YES/NO] 

For this rating to be a “YES, ’’ a timeline or date of completion must 
accompany every step/component of improvement plans. For example, the 
workshops will be provided in the summer of 2005 and the district will 
hire additional instructional assistants for the 2005-06 school year. 

c. Improvement plans include a description of data or infonnation to be used to 
evaluate success of the plan. [YES/NO] 

For this rating to be a “YES, ” improvement plans must describe the data or 
information that will be used to determine whether the steps/components of 
the improvement plan addressed the problem area. For example, scores from 
the state-wide reading assessment will be used to determine if summer 
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reading workshops and the hiring of additional instructional assistants 
addressed the problem of low reading scores. 



F3. Defensible Improvement Plans 

Improvement plans reflect research and evidence-based practices, identify support 
for implementation, and include oversight. 

General Guidelines: You should review a template for improvement plans or 
examples of LEA/EIS program improvement plans to determine whether the state 
requires plans to include descriptions of the research and data on which they 
were based. The plans should also describe how the state will support 
implementation and how the state will monitor implementation of improvement 
plans. Written documentation is required to rate the following elements. The 
interview with the monitoring director may provide supporting information, 
particularly related to research and evidence-based practices. 

a. Improvement plans include properly cited supporting infonnation and data 
(e.g., evidence-based practices). 

1 . Improvement plans are supported by data from the problem investigation 
process. [YES/NO] 

For this rating to be a “YES, ” improvement plans must include a 
discussion of how data or findings from the problem investigation process 
were used in developing the improvement plan. For example, when 
investigating low reading scores, the state discovers that the schools with 
the lowest scores had the highest number of students on the free and 
reduced lunch program, and therefore the plan focused specifically on 
these schools. 

2. Improvement plans include documentation of research and evidence-based 
practices on which the plans are based. [YES/NO] 

For this rating to be a “YES, ” improvement plans must include a 
discussion about how best available empirical evidence was used when 
developing improvement plans. For example, a well-known researcher 
conducted seminars during the past year on early intervention in natural 
environments and this information provided the basis for the improvement 
plan. 

b. Plans outline intended support for LEAs/EIS programs. 

1. Plans specify TA to support implementation. [YES/NO] 

For this rating to be a “YES, ” improvement plans must outline the TA 
provided to LEAs/EIS programs to support implementation of the plan. 
For example, the state may provide training for LEA/EIS program staff or 
hire a consultant to work with the LEA/EIS program. 
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2. The plan specifies financial support for implementation. [YES/NO] 

For this rating to be a “YES, ” improvement plans must specify how 
steps/components of the plan will be financed. For example, the plan may 
indicate that professional development funds will be used for summer 
workshops or funds from the state’s lottery program may be used to hire 
additional staff 

c. Plans describe the ways that implementation will be monitored by the 
SEA/lead agency. [YES/NO] 

For this rating to be a “YES, ” improvement plans must specify how the 
SEA/lead agency will oversee or monitor implementation of the plans. For 
example, LEAs/EIS programs may be required to submit quarterly reports to 
the SEA/lead agency describing progress or the SEA/lead agency may monitor 
implemen tation through periodic phone calls with LEAs/EIS programs. 

F4. Dissemination of Improvement Plans to Stakeholders 

Improvement plans are accessible and available to stakeholders. 

General Guidelines: Interview the monitoring director to determine how 
stakeholders are made aware of improvement plans. Also, determine the extent to 
which the plans are discussed with LEAs/EIS programs and other stakeholders. 
You should further explore accessibility and availability by confirming reported 
dissemination means (such as examining websites) and through interviews with 
LEA/EIS program staff or other stakeholders. Written documentation is not 
required to rate the following elements. 

a. Improvement plans are disseminated to local stakeholders beyond the 
LEA/EIS program administrators. [YES/NO] 

For this rating to be a “YES, ” the state must require that improvement plans 
be formally disseminated to local stakeholder groups (local coordinating 
councils or advisory panels, working groups, etc.). For example, this could 
mean that the state posts all improvement plans on its website or that the state 
specifies a process that the LEA/EIS program must use to disseminate 
improvement plans. Note: If improvement plans are available upon request, 
this is NOT a formal dissemination process and the rating would be “NO. ” 

b. Opportunities are provided for LEA/EIS program personnel to discuss 
improvement plans with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ’’ the monitoring director must indicate that 
LEA/EIS program personnel are given opportunities to discuss improvement 
plans with state agency staff. These opportunities need not be formal 
opportunities; that is, they may include e-mails or telephone conversations 
with SEA/lead agency staff or exit interviews or conversations at the end of 
site visits. 
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c. Opportunities are provided for local stakeholders to discuss improvement 
plans with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must indicate that local 
stakeholder groups are given opportunities to discuss improvement plans with 
state agency staff. These opportunities need not be formal opportunities ; that 
is, they may include e-mails or telephone conversations with SEA/lead agency 
staff 

G. Improvement Plan Implementation 

The dichotomous scale (YES/NO) is applied to the elements under Improvement Plan 
Implementation because it is assumed that the state has a process for supporting 
improvement plan implementation at the local level and for monitoring progress on 
improvement plans. Remember, you are making ratings based on the state process and 
not each individual LEA/EIS program improvement plan. Also, your ratings should not 
pertain to instances of state-wide improvement planning. 

Gl. Plans and Resources for Implementation 

Resources and support are provided for implementation of the improvement plan. 

General Guidelines: Interview the monitoring director to gather information 
about what resources and supports the state provides to LEAs/EIS programs for 
implementation of improvement plans. Also, determine the extent to which the 
state has a process for monitoring progress toward implementation. Further 
explore resources and supports through interviews with LEA/EIS program 
directors. Written documentation is not required to rate the following elements. 

a. SEA/lead agency meets with LEAs/EIS programs to help put the improvement 
plan into action. 

1 . Meetings are held with SEA/lead agency to discuss local implementation. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must confirm that 
regular meetings are held with LEA/EIS program staff to discuss the 
implementation of improvement plans. Meetings do not necessarily need to 
be in-person meeting, but may include conference calls as well. 

2. LEAs/EIS programs provide regular updates to the SEA/lead agency on 
implementation progress (report, letter, memo, meeting, conference call, 
etc). [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe how 
LEAs/EIS programs provide information to the SEA/lead agency 
regarding implementation of improvement plans. For example, LEAs/EIS 
programs may be required to submit quarterly reports or participate in 
monthly calls to describe what steps/components of the plan have been 
implemented. 
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b. SEA/lead agency provides resources to LEAs/EIS programs that allow them 
to obtain additional technical assistance, training, supplemental data, funding, 
etc. related to improvement plan implementation. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe the 
resources available to LEAs/EIS programs as they implement improvement 
plans. For example, state discretionary funds will be funneled to the LEA/EIS 
program to hire a consultant or pay for attendance at a regional or national 
conference. 

G2. Fidelity of Implementation 

Plans are fully and evenly implemented in LEAs/EIS programs. 

General Guidelines: Interviews with the monitoring director should confirm that 
the state has a process to determine the extent to which improvement plans are 
actually implemented. Review and documentation that relates to improvement 
plan implementation. Written documentation is not required to rate the following 
elements, except where noted. Further explore improvement plan implementation 
through in terviews with LEA/EIS program directors. 

a. Documentation confirms that plans were followed or changes in plans 
justified. [YES/NO] 

For this rating to be a “YES, ” you must review written documentation that 
denotes whether the activities described in the improvement plan were 
actually carried out. For example, the improvement plan might have a place 
to indicate the date the activity was completed or dates of completion might be 
noted in progress reports submitted to the SEA/lead agency. If the activity was 
not carried out, the documentation should include a reason why and, if 
appropriate, propose an alternate activity. 

b. SEA/lead agency provides follow-up and feedback to LEAs/EIS programs on 
a regular basis concerning their progress in implementing improvement plans. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must be able to 
describe how feedback is provided to LEAs/EIS programs regarding the 
implementation of improvement plans. For example, if the LEA/EIS program 
submits a report to the state detailing progress made, then the state may 
provide oral or written feedback regarding the report. The state may also 
follow-up with LEAs/EIS programs via regularly scheduled conference calls, 
during which the state provides feedback about progress on improvement 
plans. 
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Component: Reassessment 



H. Reassessment 

HI. Defensible Findings 

The findings drawn from the reassessment process are adequately defended and 
supported by comparing LEA/EIS program performance to specified targets. 

General Guidelines: Remember that reassessment is conducted in the context of 
an improvement plan or CAP that has been implemented. Reassessment of 
performance/compliance on the specified indicators is adequately defended and 
supported by comparing LEA/EIS program performance/compliance to specified 
targets after corrective actions or improvement plans have been implemented. 
Defensible findings relate to the quality of the discussion on the reassessment 
findings, not to the specific components of (or existence of the report on the 
reassessment process, i.e., Complete Reporting. 

Review documentation to assess the extent to which findings from the 
reassessment process are defensible. The state should have guidelines or a 
template for reporting on findings from the reassessment process. Written 
documentation is required to rate the following elements. Written documentation 
can take the form of a report, memo, letter, PowerPoint presentation, or similar 
documentation. If no written documentation is available, the following elements 
should be rated “NO. ” 

a. Findings reflect performance in relation to specific targets. [YES/NO] 

For this rating to be a “YES, ” the findings must discuss LEA/EIS program 
performance in relation to targets. The targets to which performance is being 
compared may be the same targets that were used to initially identify a 
problem, or they may be new targets that have been specified in an 
improvement plan or CAP. For instance, if an LEA/EIS program has 
exceedingly low performance on a specified target in the problem 
identification stage, the desired level of performance after an improvement 
plan or CAP has been implemented may be progress toward the state-level 
target. 

For example, suppose a state has an indicator on the percentage of students 
who pass the eighth grade statewide reading assessment and the target is 
84%. Only 25% of students have passed the assessment in a particular LEA. 
An investigation is conducted to determine why performance is so low in this 
LEA and an improvement plan is developed and implemented to increase 
performance on this indicator over time. The plan may set a new target that 
the percentage of students who pass the eighth grade statewide reading 
assessment should be 50% after the first year of plan implementation. The 
improvement plan may specify that performance. 
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b. Findings are adequately supported by data. [YES/NO] 

For this rating to be a “YES, ” reassessment findings must be supported by the 
data. If no data are presented then the reviewer must assume the discussion is 
NOT adequately supported. 

H2. Complete Reporting 

Reassessment reports completely describe the reassessment process so that 
essential infonnation is provided and easily accessible to LEAs/EIS programs and 
other stakeholders. 

General Guidelines: At a minimum, reassessment should include evaluation of 
LEA/EIS program progress on the original performance/compliance indicators 
and address progress toward the original targets. Reassessment reports may be 
written by state staff or local-level staff. In either case, the state should have 
guidelines or a template for reporting. Review documentation to assess whether 
the reassessment process is described completely, whether the report includes all 
essential information, and whether the report is accessible to stakeholders. 
Written documentation is required to rate the following elements, except where 
noted below. Written documentation can take the form of a report, memo, letter, 
PowerPoint presentation or similar documentation. If no report exists, all the 
following elements should be given the rating “NO. ” 

a. Reports focus on specified indicators and targets. [YES/NO] 

For this rating to be “YES, ” the report must discuss LEA/EIS program 
performance on the indicators and targets, and include a presentation of 
findings associated with the performance of LEAs/EIS programs on the 
indicators after an improvement plan or CAP has been implemented. Note 
that each finding should be associated with an indicator/target (either the 
original target or a new target specified in the improvement plan/CAP). 

b. Reports describe the data used to reassess performance/compliance on 
specified indicators and targets. [YES/NO] 

For this rating to be “YES, ” the report must describe the data used to arrive 
at the level of performance or compliance. At a minimum, the report presents 
a discussion of the nature (e.g., survey, observation) and original source of 
the data (e.g., student, teacher). 

c. Reports describe the methods and procedures used to reassess performance or 
compliance. [YES/NO] 

For this rating to be “YES, ” the report must include a discussion of the 
methods and procedures used to arrive at a comparison of the performance(s) 
to the target(s). At a minimum, the report should describe the general data 
collection strategy and the process by which data are aggregated and 
compared to the target. 
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d. Reports describe limitations of the data used to reassess performance or 
compliance. [YES/NO] 

For this rating to be “YES, ” the report must include a discussion of the 
limitations of the data. For example, the report may discuss the accuracy, 
validity, and reliability of the data collection instruments. If sampling is used, 
then there may be a discussion of how representative the sample is and 
whether the findings can be generalized. 

e. Visual displays are clear and understandable and relate to the findings/data. 
[YES/NO/NA] 

For this rating to be a “YES, ” visual displays should be straightforward 
(accurately labeled) and offer the reader a clear understanding of the 
findings/data. 

A rating of “NA ” means there were no visual displays of data in the report. 

f. Reassessments of LEA/EIS programs include a discussion of strengths and 
areas needing further improvement. [YES/NO] 

For this rating to be a “YES, ” the report must address the strengths and areas 
still needing improvement (with regard to the indicators for which 
performance is reassessed). 

g. Someone who did not write the report or collect the data reviews the reports. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must confirm that an 
independent reviewer was used in the report preparation process before the 
report was disseminated to an intended audience. That is, someone other than 
the report’s author or participant in the data collection process read the 
report and provided feedback on its clarity and completeness. Written 
documentation is not required to rate this element. 

H3. Dissemination of Reassessment Reports to Stakeholders 

Reassessment reports are accessible and available to stakeholders. 

General Guidelines: Interview the monitoring director about the availability of 
reassessment reports to LEA/EIS program staff and other stakeholders, and 
whether discussions occur between stakeholders and SEA/lead agency staff. 
Further explore accessibility and availability by confirming what reported 
dissemination means (such as examining websites) and through interviews with 
stakeholders and local district/ program staff. 

a. Reassessment reports are disseminated to local stakeholders beyond the 
LEA/EIS program administrators. [YES/NO] 

For this rating to be a “YES, ” the state must require that reassessment 
reports be formally disseminated to local stakeholder groups (local 
coordinating councils or advisory panels, working groups, etc.). For example, 
this could mean that the state posts all reassessment reports on its website or 
that the state specifies a process that the LEA/EIS program must use to 
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disseminate reassessment reports. Note: If reassessment reports are available 
upon request, this is NOT a formal dissemination process and the rating 
would be “NO. ” 

b. Opportunities are provided for LEA/EIS program personnel to discuss 
reassessment reports with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must indicate that 
LEA/EIS program personnel are given opportunities to discuss reassessment 
reports with state agency staff. These opportunities need not be formal 
opportunities; that is, they may include e-mails or telephone conversations 
with SEA/lead agency staff or exit interviews or conversations at the end of 
site visits. 

c. Opportunities are provided for local stakeholders to discuss reassessment 
reports with SEA/lead agency staff. [YES/NO] 

For this rating to be a “YES, ” the monitoring director must indicate that local 
stakeholder groups are given opportunities to discuss reassessment reports 
with state agency staff. These opportunities need not be formal opportunities ; 
that is, they may include e-mails or telephone conversations with SEA/lead 
agency staff. 

d. Findings from reassessments are disseminated to state-level stakeholders. 
[YES/NO] 

For this rating to be a “YES, ” the monitoring director must describe actions 
taken to inform state-level stakeholders, including the state-level stakeholder 
committee, about reassessment findings. The state may disseminate individual 
reassessment reports or develop a report that summarizes results of 
reassessments across LEAs/EIS programs. 



C-39 




Appendix D 

Site Visit Documentation Checklist 



Below is a list of documentation that the states were asked to provide in advance of the site visit. 
Having this information in advance provided the site visit teams with a general understanding of 
the states’ Part B and Part C monitoring systems and allowed the site visit teams to better focus 
their efforts during the site visits. 

States were asked to provide documentation pertaining to the timeframe on which the site visit 
focused (i.e., 2004-05 for the first round of site visits and 2006-07 for the second round of site 
visits). The documentation could have been in various formats, including manuals, memos, 
reports, report templates, PowerPoint presentations, etc. If any of this documentation was 
available on the states’ websites, states were asked to provide a specific URL (and not just the 
state’s general website address). The project team requested, however, that states not include any 
documentation for the site visit teams to review that contained confidential information, such as 
child names, unique child identifiers, or Social Security numbers. 

The project team was aware that there may have been some documents on this list that contained 
information that was not routinely made public and that states could not send these documents in 
advance without following proper clearance procedures. Therefore, the team told states that the 
site visit teams could review these types of documents while on site, and asked states to make 
them available to the site visit team then. 

□ General policies and procedures related to the monitoring and improvement process. 

□ List of stakeholders that participate in any aspect of the monitoring and improvement 
process. 

□ List of the indicators and targets used in monitoring local education agencies 
(LEAs)/Part C programs. 

□ Procedures for selecting LEAs/Part C programs for monitoring activities. 

□ Descriptions of monitoring activities (e.g., on-site visits, self-assessments, etc.). 

□ Policies and procedures related to monitoring data, including: 

□ Sources of data; 

□ Processes for collecting and analyzing data; 

□ Confidentiality and protection of human subjects; and 

□ Templates of reports related to monitoring findings. 
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□ Documents related to corrective action, including: 

□ Templates for corrective action plans; and 

□ Written procedures regarding the enforcement of corrective actions, including 
any sanctions that may be used by the state. 

□ Documents related to improvement planning, including: 

□ Templates for improvement plans; and 

□ Written procedures regarding the implementation of improvement plans. 
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Appendix E 

Site Visit Interview Probes 



This appendix presents interview probes that were provided to the site visitors to use when 
conducting site visit interviews. These probes were developed to help site visitors gather and 
organize the infonnation needed to understand the nature and design of states’ monitoring 
systems and to rate whether the site visit rating elements were present in those systems. 

The interview probes were somewhat general because of the variation that exists in state 
monitoring systems. Site visitors needed to ask specific follow-up questions. In addition, site 
visitors may have needed to probe out of sequence, so they tailored their questions to the 
responsibilities of the person whom they were interviewing. For example, the person supervising 
monitoring may have been able to tell the site visitor how the state decided upon specific 
indicators and how problems are identified but might not have been able to answer specific 
questions about the data collection process. Site visitors may have needed to save those questions 
for the data manager or for one of the other staff members who worked more closely with the 
data. 

Site visitors were instructed to begin the site visit with an interview with the Part B state director 
or the Part C coordinator, during which they asked him/her to provide a brief overview of the 
state’s monitoring system. If the person in the state with the most knowledge of the monitoring 
process was not the aforementioned person, the initial interview should have included both the 
Part B state director/Part C coordinator and the monitoring supervisor/director. General topics 
included a description of the indicators and targets the state used to monitor, the process used to 
set those indicators and targets, stakeholder involvement in the state’s monitoring process, the 
process used to collect data for measuring performance on those indicators and targets, 
identification of problems, further investigation of those problems, and the process used to 
correct problems (i.e., Corrective Action and Enforcement; Improvement Planning and 
Implementation). If these individuals did not know the answers to the questions, they were asked 
to direct the site visitor to the person who would have that information. 

The site visit teams conducted interviews with additional state agency staff involved with 
monitoring activities, local education agency (LEA) and early intervention services (EIS) 
program staff members, and stakeholders to garner more specific details on various state 
monitoring and improvement activities. The site visit teams also reviewed any relevant written 
documentation the state could provide on monitoring activities to gain additional insight on a 
state’s monitoring system, as well as to confirm verbal descrip tions/explanations provided during 
interviews with staff. 



Interview Probes for State Agency Staff 

The interview probes for the state agency staff were organized by the components from the 
framework for monitoring. 
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Problem Identification 



1 . Where can I get a list of the indicators your state used in 2006-07 and their definitions? 

2. Does your state use additional indicators other than those listed? 

3. How did your state arrive at these particular indicators/targets? 

4. Who was involved in this process? 

a. Were stakeholders involved? If so, what types? 

b. How were they selected? 

c. What roles did these individuals play? 

5. To what extent were the following types of individuals aware of the indicators/targets? 

a. General education leadership (e.g., state superintendent, state school board 
members)? (Applies to Part B only.) 

b. Governmental figures (e.g., representatives from the governor’s office, representative 
from education/health committee of state legislature)? 

c. Part C (or Part B if visiting a Part C program)? 

6. How did your state go about identifying targets for each indicator? 

7. What is the data collection cycle (e.g., annual, every 2 years)? 

8. Do you collect data from all districts/programs? If not how do you decide which 
districts/programs to collect data from (including charter schools)? 

9. What is the process for collecting data to determine district/program performance on the 
indicators (e.g., site visits, district/program self-assessments, desk audits)? 

a. What are the sources of the data (e.g., child/student records, 618 data, statewide 
testing data, interviews with students/teachers, classroom observations)? 

b. Who collects the data (e.g., monitoring teams, contractors, service providers, 
teachers, administrators)? 

c. Who oversees the data collection effort at the state level? 

d. Is this process documented in any memos, manuals, etc.? 

10. What sort of information (memo, manual, etc.) or training do data collectors receive? 

a. Is it specific to the data being collected? 
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b. What about general data collection issues, such as accuracy, validity, and 
confidentiality? 

11. Does the state have any directives addressing protection of human subjects? 

12. How does your state ensure the quality of the data being collected? 

a. Are the data coded and scored? Are the procedures for doing this documented? 

b. What about data provided directly by district/program? Do you have procedures to 
verify the quality of these data? 

13. How are data protected and stored? 

14. Who has access to the data? 

15. Is sampling used to reduce the data collection burden? 

16. How are the collected data used to assess district/program performance on the indicators? 

17. What is the process for informing districts/programs that they are not meeting targets for 
specific indicators or that they have noncompliance issues? 

a. Are reports issued? (Ask to see a template/example if one is available.) 

b. What information is contained in these reports? 

c. Who receives copies of these reports? 

d. Are there opportunities for those who receive the reports to discuss them with you or 
your staff? 

18. Are any of your stakeholders informed about district/program performance? 

a. When and how are they informed? 

b. Are they ever provided copies of actual reports that are issued? 

Problem Investigation 

1. Once you have detennined that a district/program has an issue with performance or 
compliance, do you take any steps to find out more about that issue and why it exists? 

2. If it seems like the state may do problem investigation, then probe more by asking: 

a. What steps are taken? 

b. Who is involved in the process? 
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• In what levels and ways are they involved? 

• Do stakeholders play a role? 

c. What kind of training or guidance is provided to those involved in the process? 

d. Are any of these procedures documented? 

e. Are additional data collected during this process? 

• What are the sources of the data (e.g., child/student records, 618 data, 
statewide testing data, interviews with students/teachers, classroom 
observations)? 

• Who collects the data (e.g., monitoring teams, contractors, service providers, 
teachers, administrators)? 

• How are the accuracy and overall quality of data assessed? 

• How were the data analyzed? Who performs the analyses? 

f. How are the findings from this process documented? Are they incorporated into any 
monitoring reports? 



Corrective Action and Enforcement 

1 . When do issues of noncompliance lead to the development of corrective action plans? (Ask 
to see a template/example if one is available.) 

a. Who develops the corrective action plan (e.g., the monitoring team, state staff, 
district/program staff)? 

b. What kinds of information are included in the corrective action plan? 

c. Are there opportunities for district/program staff to discuss corrective action plans 
with you? 

2. How does the state monitor progress on correcting noncompliance? 

3. Are corrective action plans made available to stakeholders and the public? If so, how (e.g., 
stakeholder meetings, posted on a website, if requested)? 

4. What enforcement procedures are available to you if the district/program fails to correct the 
noncompliance? 

a. At what point are these procedures used? 

b. Are these procedures documented anywhere? 
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Improvement Planning and Implementation 

1 . What about issues that are not compliance related or that are more systemic in nature? Are 
improvement plans developed to address these types of issues? 

2. If it seems like the state may do improvement planning, then probe more by asking: 

a. What triggers the need for an improvement plan? Who makes the decision that one is 
needed? 

b. Who is involved in the improvement planning process (e.g., the monitoring team, 
state staff, district/program staff, stakeholders)? 

c. How are improvement plans developed? 

d. What kinds of information are included in improvement plans? (Ask to see a 
template/example if one is available.) 

3. When is a plan considered ready for implementation? Who has to sign off on it? 

4. What is the state’s role in helping districts/programs implement improvement plans (e.g., 
meetings with local staff, technical assistance, resources)? 

5. How is the implementation of the improvement plans by district/programs monitored (e.g., 
phone calls, progress reports)? 



Reassessment 

1 . How do you detennine if the corrective action plans and improvement plans that were 
implemented had or are having the desired effect? 

2. If it seems like the state may do reassessment, then probe more by asking: 

a. Are any additional data collected? 

• If so, what kind? 

• How often are they collected? 

• How are they analyzed? 

b. Who is involved in this process? 

c. Does this occur as part of the next monitoring cycle or is it a separate process? 

d. Are the findings documented in any type of report? (Ask to see an example if one is 
available.) 
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• What kind of information is included in these reports? 

• Who receives these reports? 

• Are there opportunities for those who receive the reports to discuss them with 
you or your staff? 

• Are findings shared with stakeholders? 



Interview Probes for Additional Interviews 

During the course of the site visit, site visitors also conducted interviews with local staff 
members (i.e., staff from LEAs/EIS programs that participated in the monitoring activities) and 
other stakeholders (i.e., parents or advocates who served on the state-level stakeholder 
committee or participated in the monitoring activities). The purpose of these interviews was to 
confirm some of the infonnation that the site visitors heard from the state agency staff. Thus, site 
visitors were instructed to conduct these interviews on the second day of the site visit, if possible, 
after they gained a good understanding of the state’s monitoring system and the roles these 
individuals may have played. 



Interview Probes for Local Staff 

1 . How were you involved in the state’s monitoring and improvement activities in 2006-07? 

2. What was the process for identifying issues with district/program performance or issues with 
noncompliance? 

a. Were there reports that documented this process and issues that were found? 

b. What kinds of information were contained in these reports? 

c. Were there opportunities for you to discuss these reports with the state staff? 

3. When did issues of noncompliance lead to the development of corrective action plans? 

a. Who developed the corrective action plan (e.g., the monitoring team, state staff, 
district/program staff)? 

b. What kinds of information were included in the corrective action plan? 

c. Were there opportunities for you to discuss corrective action plans with the state 
staff? 

d. To what extent were corrective action plans made available to stakeholders and the 
public? 
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4. What about issues that are not compliance related or that are more systemic in nature? Were 
improvement plans developed to address these types of issues? 

a. Who was involved in the improvement planning process (e.g., the monitoring team, 
state staff, district/program staff, stakeholders)? 

b. How were improvement plans developed? 

c. What kinds of information were included in improvement plans? 

d. Were there opportunities to discuss improvement plans with state staff? 

e. What was the state’s role in helping districts/programs implement those plans (e.g., 
meetings with state staff, resources/technical assistance/training)? 

f. Did the state follow up with you regarding the implementation of the improvement 
plans (e.g., phone calls, progress reports)? 

5. How did the state determine that corrective action plans and improvement plans were 
successful? 

a. Was a report written? If so, by whom? 

b. Were there opportunities for you to discuss these reports with state staff? 



Interview Probes for Stakeholders 

1 . How were you involved in the state’s monitoring and improvement activities in 2006-07? 

2. Were you involved in detennining what indicators/targets would be used to monitor 
district/program performance? 

a. How were you involved? 

b. Who else was involved in this process? 

c. Did the state provide you with materials to help you understand this process and what 
they were asking you to do? Did you find these materials to be helpful? Was there 
something else that the state could have done that you would have found more 
helpful? 

3. If the state engaged in Problem Investigation, then ask: Once an issue was identified for a 
particular district/program, did you participate in the process for finding out more about that 
issue and why it existed? 

a. Can you describe this process? 

b. How were you involved? What role did you play? 
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c. Who else was involved in this process? 

4. When the state identified noncompliance issues for a particular district or program, were the 
corrective action plans made available to you or to other stakeholders? 

a. If so, how were they made available? 

b. Were there opportunities to discuss corrective action plans with state or local staff? 

5. If the state engaged in Improvement Planning, then ask: Were you involved in the 
improvement planning process? 

a. How were you involved? What role did you play? 

b. Did the state provide you with materials to help you understand this process and what 
they were asking you to do? Did you find these materials to be helpful? Was there 
something else that the state could have done that you would have found more 
helpful? 

c. Who else was involved in the improvement planning process at the state level and at 
the district/program level? 

d. Were improvement plans made available to you or to other stakeholders? 

e. If so, how were they made available? 

f. Were there opportunities to discuss corrective action plans with state or local staff? 

6. How did the state determine that corrective action plans and improvement plans were 
successful? 

a. Was a report written? If so, by whom? 

b. Were these reports made available to you or to other stakeholders? 

c. If so, how were they made available? 

d. Were there opportunities to discuss these reports with state or local staff? 
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Appendix F 

Reliability of the Site Visit Data Collection 



As discussed in chapter 4, the project team conducted a reliability study to determine if the site 
visit teams consistently rated each state’s Part B and Part C monitoring processes. This appendix 
presents more detailed information about the reliability study. 



Overview 

It would be too costly — and intrusive to the states — to send two separate teams on site visits, 
have them observe all interviews and review all documents, and then develop ratings that could 
be compared for purposes of reliability. Thus, the project team used a less intrusive method. The 
project team sampled the completed site visits from eight states and then randomly selected two- 
person comparison teams to develop new ratings. The comparison teams did not make actual site 
visits. Instead, they used audio recordings of the original site visits and available documentation 
to complete their ratings. The ratings of the site visit team and the comparison team were then 
compared to establish an “inter-team” reliability. 



Steps Taken to Assess Reliability 

The reliability study was conducted using four steps. This section describes each of these steps in 
detail. 

Step 1: Select Eight Site Visits 

The project team randomly selected four Part B and four Part C site visits from the list of original 
site visits. The randomly selected sites included large and small states as well as states 
representing each region of the country. 

Step 2: Assign Comparison Teams to Each Site Visit 

Two-person comparison teams were randomly selected from the pool of trained site visitors. 
First, the original site visit team members were removed from the pool; then, two people were 
randomly selected to form a comparison team. Each time a comparison team was fonned, the 
team members were removed from the eligible pool. 

Step 3: Assemble and Disseminate Site Visit Documentation and Audiotapes 

Project staff assembled the audiotapes and documentation from each site visit and sent copies to 
each member of the comparison team for the state to which they were assigned. It should be 
noted that while the comparison teams had access to the audiotapes and documentation, they did 
not have access to the original site visit team ratings or reports, nor were they allowed to confer 
with the original site visit team during the reliability study, to ensure that the comparison team 
ratings were developed independently. 
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In a limited number of cases, the comparison team did not have access to all of the documents 
made available to the original site visitors during the site visit. Unfortunately, some states would 
not allow particular materials to leave the premises. The missing documents typically affected 
only a small number of ratings. In these cases, the comparison teams were asked to use all of the 
information that was available to them to infer the relevant ratings as best they could. While less 
than optimal, the overall results do provide a valid measure of the consistency of ratings across 
site visits. 

Step 4: Complete a Single Set of Team Ratings 

Many state monitoring systems included more than one monitoring process. In an effort to 
reduce burden on the comparison teams, the original teams each identified one monitoring 
process to be rated by the comparison team. The selected monitoring process then became the 
focus of the comparison team’s rating effort. To ensure that the original team did not select what 
they considered to be the easiest process to rate, which would have introduced bias into the 
reliability study, the original site visit team selected the process that included the most 
components of the framework for monitoring. If more than one process included the same 
number of components, the original site visit team decided which one to use for the reliability 
study by determining which of these processes they considered to be the state’s primary 
monitoring process. 

All of the comparison teams used the same rating procedures. That is, team members reviewed 
the audiotapes and documentation independently within an agreed-upon 5-day period. Then, as 
for the site visits, each team member independently rated each element. The team members then 
discussed their initial ratings and noted any elements on which they disagreed. If there was a 
disagreement, the team members went back to the interviews and documentation and discussed 
the element until they could come to an agreed-upon rating. These steps were repeated until the 
two team members had a final single set of ratings with which they both agreed. Because of the 
iterative process used to develop the ratings for both the site visits and the reliability study, it was 
this final agreed-upon set of ratings that was then compared with the ratings made by the original 
site visit team. The same process was used for the second round of site visits. 



Results 

Technically, the process of comparing the ratings of the site visit team and the comparison team 
involves establishing an “inter-team” reliability. While a simple percentage of agreement 
between the two teams could have been calculated, this technique does not correct for the 
possibility that agreement on some ratings could have occurred purely by chance. (This seemed 
likely given that there were dozens of site visit rating elements.) To overcome this problem, the 
intraclass correlation coefficient (ICC), which corrects for chance, was calculated. 

First, the data were aggregated to give an overall score for each set of raters across the five 
components. An ICC was then calculated for the total score. The result for the 2004-05 reliability 
study data was .84. The result for the 2006-07 reliability study data was .77. Acceptable levels of 
reliability are typically considered to be values greater than .70 (see Stemler 2004). 
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Appendix G 

Description of Rasch Analysis 



To reduce the burden on staff interviewed, the project team sought to reduce the 137 items that 
made up the original site visit rating protocol used in first round of site visits to a more 
manageable and efficient set of items within each framework component. The strategy used to 
develop the measures and reduce the data collection burden was Rasch analysis, which is 
described in more detail below. The goal was to identify the elements that would yield a single 
reliable continuous measure for each framework component. This appendix describes the process 
used to arrive at a reliable set of measures and includes a brief introduction to the Rasch 
procedure and how the Rasch procedure was applied to the first round of site visit data. 1 



Rasch Analysis 

The main objective of a Rasch analysis is to quantify the value of an unobservable latent trait. 
While the latent trait is only indirectly measurable, it is assumed that it can be constructed from 
the ratings of the elements for the particular monitoring component: Problem Identification, 
Problem Investigation, etc. The probability that a latent trait exists is a function of a given state's 
position on the latent trait plus certain parameter(s) associated with a particular item. 2 Given 
ratings for many elements (e.g., the 64 elements that make up the Problem Identification 
component), Rasch modeling provides a way to estimate the subject (i.e., individual state 
process) and item parameters using individual state rating patterns. This probability is modeled 
by means of logistic models. 

The Rasch approach to scale development offers unique properties of scales not easily obtained 
from other measurement approaches. The Rasch model meets two commonly held assumptions 
underlying scale construction: (1) each item contributes equally to the measure of the particular 
trait, and (2) each item is measured on the same equal interval scale. Neither of these 
assumptions is true with traditional scale construction techniques. With the use of logistic 
models, it is possible to better estimate the distances between scale points, and to add responses 
to items together since the scales use the same metric. An item may be measured on a Likert-type 
scale, e.g., 0 implying poor quality, 1 implying good quality, 2 implying excellent quality, and so 
on. In this instance, the space, for example, from 0 to 1 and from 1 to 2, is not necessarily 
equidistant. Rasch analysis makes it possible to establish equal units on the item scale, thus 
revealing the true distance(s) between scale points. Similarly, two different questions, each 
having a yes/no response option may yield different distances between the 0 and 1 when 
analyzed with the Rasch technique. When differences are found, they suggest that additivity of 



1 Appendix F discusses the reliability of the site visit data collection. Inter-team reliability is an estimate of how 
precise the ratings of the site visit teams were. Appendix G discusses creating reliable measures for each of the 
study framework components. In this context, reliability refers to how precise the different items were in 
measuring an underlying trait. The two types of reliability address two sources of error in the measurement 
process: a) the disagreement of raters and b) the lack of consistency between the responses to items within the 
same scale. 

2 The term “item” will be used throughout when referring to ratings of the individual elements made by site visitors. 
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the original scale points is misleading and inappropriate. With analysis of the scale points on a 
common metric (such as logits), one can readily assess the contribution of the item (as a whole) 
to the latent trait. 



The Rasch Model 

The analysis used the partial credit model as opposed to the rating scale model that assumes a 
common scale for each item (Bond and Fox, chapter 7, 2001). The partial credit model allows for 
different scaled items to be incorporated into the same model (0, 1 vs. 0, 1,2, in this case). It has 
the following sets of parameters: 



i. the position of subject 



i = !,...« 



on the monitoring quality continuum, and 



li. 



'li- 



the (base) position (“difficulty”) of the k-th step for item J ' ’ ' " s on the monitoring 
quality continuum. 

V — § 

The quantity ' ' jk reflects the difference between a state’s rating and item frequency. 
Suppose item j has m + 1 response levels. Then, the probability of the ith subject responding in 
category / to item j equals: 



exp 
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i jk 
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l = 0,1, • • • m, j = 1, • • • s, i = 1, • • • n 



The model is characterized by separable parameters: (1) subject (subjects in this case being state 
processes) and (2) item. Parameter separability makes it possible to (statistically) condition the 
subject parameters out of the item calibration, resulting in sample-free calibration. Similarly, the 
item parameters can be conditioned out of subject (i.e., individual state process) measurement, 
resulting in test-free measurement. Thus, the state monitoring process implementation and item 
position frequency on the continuum can be estimated independently by means of conditional 
maximum likelihood estimation. This means that the state quality and the item quality scores are 
statistically sufficient for estimating implementation measures and item calibrations. 



Infit and Outfit Statistics 



The two key statistics used in the scale development process are infit and outfit statistics. Their 
primary purpose is to assist in identifying poor fitting items. The infit statistic is a weighted fit 
statistic that is sensitive to unexpected responses to items near a state’s expected perfonnance 
level. The outfit statistic is an outlier-sensitive fit statistic that is sensitive to unexpected 
responses (i.e., ratings) on items far from the state’s expected perfonnance level. Mathematically 
these statistics are explained as follows: 



Let the response for state z on the item J be 



x 

1J . 
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x.. 

Let the expected value of the actual response 11 be 



m 



Z kP-i 
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The residual is defined as 
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J , and the standardized residual is defined as 
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The outfit statistic is defined as outfit = — Z (Z..) , 
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and the infit statistic is defined as infit- 
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Rasch Analysis and the Site Visit Ratings Data 

The analyses were aimed at creating five more efficient measures of whether the elements 
associated with the framework were present in state monitoring processes. They included one 
measure each for Problem Identification, Problem Investigation, Corrective Action and 
Enforcement, Improvement Planning and Implementation, and Reassessment. 

Results of Rasch Analysis 

Each Rasch analysis using the five unique datasets for each of the components required multiple 
iterations before satisfactory measures of implementation were constructed. The process 
involved first examining the fit statistics, then looking at the individual item results to determine 
whether to rescale the item or eliminate it altogether, then reexamining the overall statistics. The 
final measures included the following internal consistency reliabilities: Problem Identification 
(.95), Problem Investigation (.75), Corrective Action and Enforcement (.96), Improvement 
Planning and Implementation (.93), and Reassessment (.90). In general, the results suggest 
highly reliable measures. The one exception is Problem Investigation, which still had a .75 
internal consistency reliability coefficient associated with it. While these statistics are lower than 
those for the other components, they are still quite respectable and do provide a reasonable 
measure of implementation of the Problem Investigation component of the framework (Wright 
and Masters 1982). 

Also it is important to note that a few of the items discussed below fell outside the ideal 
statistical criteria. They are noted in the text. Any time an item did not meet a statistical criterion, 
it was further reviewed for its content value to the overall intent of the component. Those few 
items mentioned below were kept in the final set of items because the project team decided that 
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they were critical items. Because of time limitations and resources, the project team did not have 
the opportunity to test new versions of the items. Therefore, team members decided to keep the 
version that was used in round one so long as the item did not reduce the overall component 
reliability to an unacceptable level. 

Problem Identification 



The final set of elements for the Problem Identification component included 23 of the original 64 
elements from the three subcomponents that make up the Problem Identification component. The 
final set of elements is presented in table G-l by measure, error, and infit and outfit statistics. 
The individual item measures range from a low of -2.09 to a high of 2.24. Only one infit statistic 
falls above the ideal of 1.50. However, two elements (C3A and C4B) have outfit statistics of 2.02 
and 2.35, respectively. While these elements lower the reliability estimate somewhat, they were 
considered critical to the content of the component. Rather than modify the elements, it was 
decided to include them in the final set as the reliability was not greatly affected by keeping 
them. 

Problem Investigation 

The final set of elements for the Problem Investigation component included 1 1 of the original 25 
elements associated with this component. The final set of elements is presented in table G-2 by 
measure, error, and infit and outfit statistics. The individual item measures range from a low of 
-1.34 to a high of 1.00. All of the infit and outfit statistics fall well within the acceptable range of 
1.50. 

Corrective Action and Enforcement 



The final set of elements for the Corrective Action and Enforcement component included 7 of 
the original 1 1 elements. The final set of elements is presented in table G-3 by measure, error, 
and infit and outfit statistics. The individual item measures range from a low of -1.21 to a high of 
2.96. Only one outfit statistic falls above the 1.50 ideal. That element (E2A) has an outfit statistic 
of 9.90. This element was included in the final set because of its contribution to the content of 
the component. Rescaling did not improve the item statistics. 

Improvement Planning and Implementation 

The final set of elements for the Improvement Planning and Implementation component included 
15 of the original 24 elements from the two subcomponents that make up the that component. 
The final set of elements is presented in table G-4 by measure, error, and infit and outfit 
statistics. The measures range from a low of -2.21 to a high of 1.39. Only one infit statistic (F4B) 
falls above the 1.50 ideal. However, it is very close to the ideal at 1.55 and was kept because of 
its contribution to the content of the component. 

Reassessment 



The final set of elements for the Reassessment component included 9 of the original 13 elements. 
The final set of elements is presented in table G-5 by measure, error, and infit and outfit 
statistics. The measures range from a low of -2.16 to a high of 2.18. Only one infit statistic falls 
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above the 1.50 ideal. However, two elements (C3A and C4B) have outfit statistics of 2.02 and 
2.35, respectively. These elements were included in the final set because of their contribution to 
the content of the component. The fit statistics for only one element are slightly outside the ideal 
range of 1.50; H3D has an infit statistic of 1.52 and an outfit statistic of 1.84 but was included 
anyway as it was considered to contribute to the content of the component. 

Table G-l. Elements, measures, errors, and infit and outfit statistics for Problem 
Identification for site visit data: 2004-05 



Element 


Measure 


Error 


Infit statistic 


Outfit statistic 


A1A1 


0.07 


0.16 


0.87 


0.73 


A1A2 


0.53 


0.16 


1.16 


0.95 


A1A3 


0.85 


0.17 


0.92 


0.81 


A1A5 


0.12 


0.16 


0.88 


0.76 


A1B 


-0.88 


0.28 


0.84 


0.41 


A1C 


0.38 


0.16 


0.87 


0.69 


A3A 


-2.09 


0.28 


0.84 


0.41 


A3B 


2.24 


0.29 


0.86 


0.79 


A4A 


0.98 


0.22 


0.86 


0.53 


A4B 


0.70 


0.21 


0.79 


0.58 


B3A2 


-1.43 


0.26 


1.04 


1.01 


B4A 


0.12 


0.16 


1.02 


0.81 


B4D 


-0.41 


0.21 


0.94 


0.85 


B4D 


-0.50 


0.22 


1.10 


1.03 


B5B 


-0.54 


0.19 


1.13 


1.03 


CIA 


-0.07 


0.18 


1.02 


0.98 


C1C 


-0.87 


0.22 


0.90 


0.79 


C2A 


0.80 


0.16 


0.97 


0.82 


C2C 


1.15 


0.18 


1.07 


1.01 


C3A 


-0.27 


0.17 


1.56 


2.02 


C4A 


1.02 


0.17 


1.00 


0.71 


C4B 


1.60 


0.21 


1.81 


2.35 


C4C 


0.96 


0.17 


1.05 


0.78 


Mean 


0.00 


0.20 


1.02 


0.97 


SD 


1.00 


0.04 


0.23 


0.43 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006. 



G-5 




Table G-2. Elements, measures, errors, and infit and outfit statistics for Problem 
Investigation for site visit data: 2004-05 



Element 


Measure 


Error 


Infit statistic 


Outfit statistic 


D1A1 


0.47 


0.25 


1.40 


1.46 


D1A3 


1.00 


0.27 


1.07 


0.91 


DIG 


-0.90 


0.41 


1.31 


0.80 


D2B 


-1.34 


0.55 


0.55 


0.12 


D3B 


-0.06 


0.27 


0.89 


0.72 


D4B1 


-0.61 


0.35 


0.72 


0.50 


D5A 


0.21 


0.26 


1.16 


1.18 


D5B 


0.47 


0.25 


0.96 


0.95 


D5C 


0.59 


0.25 


0.89 


0.94 


D6B 


0.57 


0.29 


0.64 


0.58 


D8A 


-0.40 


0.31 


1.35 


1.16 


Mean 


0.00 


0.31 


0.99 


0.85 


SD 


0.69 


0.09 


0.28 


0.35 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006. 


Table G-3. 


Elements, measures, 


errors, and infit and outfit statistics for Corrective Action 




and Enforcement for site visit data: 2004-05 




Element 


Measure 


Error 


Infit statistic 


Outfit statistic 


E1A1 


-1.03 


0.28 


0.97 


0.73 


E1B1 


-0.35 


0.21 


0.72 


0.35 


E1B2 


-1.03 


0.28 


1.01 


0.49 


E2A 


2.96 


0.31 


1.24 


9.90 


E2B 


-1.21 


0.32 


0.76 


0.22 


E3A1 


0.06 


0.20 


0.94 


0.62 


E3A2 


0.59 


0.20 


0.98 


1.07 


Mean 


0.00 


0.26 


0.95 


1.91 


SD 


1.35 


0.05 


0.16 


3.27 


SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006. 
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Table G-4. Elements, measures, errors, and infit and outfit statistics for Improvement 
Planning and Implementation for site visit data: 2004-05 



Element 


Measure 


Error 


Infit statistic 


Outfit statistic 


F1A1 


0.56 


0.21 


1.17 


3.74 


F1A3 


1.26 


0.25 


0.72 


0.43 


F2A1 


-1.19 


0.31 


0.40 


0.16 


F2A2 


-1.39 


0.33 


1.03 


0.38 


F2B1 


-0.70 


0.27 


0.86 


0.77 


F3A2 


1.39 


0.27 


0.91 


0.87 


F3B1 


-0.85 


0.28 


1.15 


1.47 


F3B2 


1.03 


0.23 


1.09 


1.01 


F3C 


0.56 


0.21 


1.15 


1.18 


F4A 


1.26 


0.25 


1.05 


0.66 


F4B 


-2.21 


0.43 


1.55 


0.87 


F4C 


1.03 


0.23 


1.30 


1.20 


G1A1 


-0.18 


0.24 


0.96 


1.04 


G2A 


-0.72 


0.29 


1.04 


0.60 


G2B 


0.15 


0.23 


0.59 


0.42 


Mean 


0.00 


0.27 


1.00 


0.99 


SD 


1.09 


0.05 


0.27 


0.81 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006. 

Table G-5. Elements, measures, errors, and infit and outfit statistics for Reassessment for 
site visit data: 2004-05 



Elements 


Measure 


Error 


Infit statistic 


Outfit statistic 


H2A 


-1.47 


0.35 


0.89 


0.63 


H2B 


-0.74 


0.28 


0.83 


0.70 


H2C 


-0.59 


0.27 


0.59 


0.51 


H2F 


0.45 


0.30 


1.09 


0.80 


H2G 


-0.74 


0.28 


1.49 


1.35 


H3A 


2.18 


0.64 


0.40 


0.06 


H3B 


-2.16 


0.53 


1.20 


1.61 


H3C 


2.18 


0.64 


0.40 


0.06 


H3D 


0.87 


0.35 


1.52 


1.84 


Mean 


0.00 


0.40 


0.93 


0.84 


SD 


1.44 


0.15 


0.40 


0.60 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006. 
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Appendix H 

Development of Component Scores 



An important step in detennining the extent to which the elements associated with the study 
framework components were present in states’ Part B and Part C monitoring processes was the 
calculation of component scores. Item Response Theory (IRT) modeling was used to calculate 
reliable component scores from the ratings assigned to each state’s monitoring processes. IRT 
modeling was used to create the component scores for two primary reasons. First, IRT not only 
took into consideration which elements were rated as present or absent, but also the frequency 
patterns of the elements. This means that, when the scores were created, some elements were given 
more weight than other elements because of their specific properties. Second, IRT was used because 
the second round of site visits used a shorter instrument than the first (i.e., 65 elements, rather 
than 137 elements). Using IRT modeling allowed the project team to apply a linking method 
(Stocking and Lord 1983) to the 65 elements that were used in both rounds of site visits to create 
a single metric. This was done so that the scores from the first round of site visits would have the 
same meaning as the scores from the second round of site visits. The remainder of this appendix 
describes how IRT modeling was used to create the component scores. 

Process for Creating Component Scores 

Creating the component scores required two steps. First, IRT modeling was used to create 
calibrated scores for each component within each set of ratings. In this instance, the term “items” 
refers to the elements rated by the site visitors. Second, the Stocking and Lord (1983) procedure 
was used to equate the two sets of calibrated data for each component. This method is commonly 
used to equate two sets of data that are assumed to measure the same construct on the same 
population but have an arbitrary difference in mean and variance. Both of these steps are 
described in more detail below. 

Calibrate the Items (Elements) That Make Up Each Component Scale 

IRT modeling was used to estimate a Part B or Part C monitoring process’s score for a given 
component. This is done by using information involving: (1) how many items for which the 
monitoring process was given credit, (2) how difficult the items were (i.e., how infrequent it was 
for a monitoring process to get credit for the item) and, (3) how strongly each item is correlated 
with the underlying construct. For example, if a monitoring process receives credit for half of the 
items, but they are the most infrequent items, then one would conclude that the monitoring 
process has a rather high score on the construct. However, if the monitoring process received 
credit for only the most common items, then one would conclude that the monitoring process has 
a rather low score on the construct. IRT scaling enables the researcher to use infonnation about 
the number of items for which a monitoring process receives credit, as well as the nature of the 
items. In this way, IRT scores give a more precise estimate of performance than, for example, 
does the simple sum resulting from adding up the total credit given for all items. This same 
reasoning holds true for items that have several levels of credit (e.g., none, some, or all) rather 
than just credit or no credit. 
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The graph in figure H-l shows the probability of receiving credit for one of the Corrective 
Action and Enforcement elements as a function of the element score. This is the “item response.” 
This graph illustrates that, as a monitoring process’s scale score increases (y-axis), it is more 
likely to get a positive rating for the element. Different elements would have a different location 
for the item response function with respect to the proficiency scale (x-axis). This indicates 
different difficulties. Also, different elements would have different slopes for the item response 
function, indicating different correlations with the proficiency. In this way the item response 
function displays all of the element characteristics. Figure H-l shows the item response for an 
element with a yes/no (i.e., 0/1) response option. The item response for an element with three or 
more response options (i.e., 0/1/2) would have response curves for every possible response. 

Figure H-l. Example of an item response function for a dichotomous element 



IRT True Score for the Corrective Action and Enforcement Scale 




Proficiency on 7-Item Corrective Action and Enforcement Scale (Theta) 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

The elements in each scale were calibrated with a combination of two-parameter logistic (2-PL) 
and generalized partial credit (GPC) item response models. The 2-PL model was used with 
binary response element scores (possible ratings of yes/no). The GPC is an extension of the two- 
parameter logistic model to the polytomous case, where elements were scored to reflect 
gradations of credit for the presence, partial presence, or absence of a given element (those 
elements with possible ratings of 0/1/2) (Muraki 1992). 

Equate the Two Rounds of Calibrated Site Visit Ratings Data 

Figure H-l illustrated the item response function for a single element. If one were to add up these 
functions for all elements in the scale, one would get a “test” characteristic curve. Figure H-2 
gives an example of a test characteristic curve that shows the total test score (y-axis) that is 
predicted for each level of proficiency (x-axis). The estimated total test score is called the model- 
based or “true” test score. The test characteristic curve is similar to the item response function in 
that the predicted outcome on the rating scale increases with the construct score. Those state 
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monitoring processes with a higher construct score will be estimated to have a higher component 
score. 

Figure H-2. Test characteristic curve for Problem Identification Scale 



IRT True Score for the Problem Identification Scale 




Proficiency on 29-Item Problem Identification Scale (Theta) 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 

Figure H-3 shows the test characteristic curves from two different ratings (the first and second 
rounds of site visits) that measure the same proficiency (Problem Identification) on the same 
population (state monitoring processes). The fact that the curves are separated indicates that the 
ratings for the first and second rounds of site visits are in two different metrics. Being in two 
arbitrary metrics, the scores on the two sets of ratings are not comparable. However, if the proper 
linear transformation is done to the construct scores of the “focal” group (the second round of 
site visits) so that the two test characteristic curves are as close as possible to each other, then 
scores on the two sets of ratings will be comparable. In this study, the Stocking and Lord 
procedure (1983) was used to equate the focal scores (the second round of site visits) to the 
reference scores (the first round of site visits) for each component. Thus, after equating, scores 
from the first round of site visits are comparable to those of the second round of site visits. 

Item parameters and scale scores for individual state monitoring processes were estimated using 
a maximum likelihood scaling procedure (Bock and Aitkin 1981). The probability of each 
monitoring process rating given the scale score can be calculated using the item response 
function like the one illustrated in figure H- 1 (note, a function for not getting credit for the item 
would also have to be included). The probability associated with monitoring process ratings 
across all elements can be combined to estimate the process’s likelihood. Similarly, the 
likelihoods for every state monitoring process can be combined to get the likelihood for the 
whole sample. The maximum likelihood estimates are the item parameters and scale scores that 
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give the largest likelihood for the sample. As a result, the reported component scores fit the data 
in the sense that they are the scores that are most consistent with how the processes were rated. 



Figure H-3. Test characteristic curves for first and second rounds of site visits on the 
Problem Identification Scale before equating: 2004-05 and 2006-07 



Test Characteristic Curves for the Problem Identification Scale 




-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 

Proficiency on 18-Item Problem Identification Scale (Theta) 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 



Transformation into Component Scores 

As figure H-4 indicates, the scale scores ranged from -6 to 6. For actual reporting, one would 
want to avoid negative values. Thus, after equating, the IRT scale scores were transformed into a 
metric that represented the estimated number of elements present for each component for each 
Part B or Part C monitoring process for each round of site visits. The correspondence between 
IRT scale score and estimated number of elements present (or true score) is given in the test 
characteristic curve for each component, as seen in figure FI-1. The estimated number present 
score was then converted to the percent present by the following: Percent Present = (estimated 
number of elements present total number of elements for that component) * 100. Therefore, 
each component score can be interpreted as the estimated percentage of elements present in a 
Part B or Part C monitoring process for a particular component. 
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Figure H-4. Test characteristic curves for first and second rounds of site visits on the 
Problem Identification Scale after equating: 2004-05 and 2006-07 



Test Characteristic Curves for the Problem Identification Scale 
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Proficiency on 18-Item Problem Identification Scale (Theta) 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Special Education 
Research, Study of States’ Monitoring and Improvement Practices Under IDEA site visit data from 2006 and 2008. 
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